[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[
https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028876#comment-14028876
]
Hadoop QA commented on HADOOP-10561:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649995/HADOOP-10561.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4049//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4049//console
This message is automatically generated.
> Copy command with preserve option should handle Xattrs
> --
>
> Key: HADOOP-10561
> URL: https://issues.apache.org/jira/browse/HADOOP-10561
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 3.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Yi Liu
> Attachments: HADOOP-10561.1.patch, HADOOP-10561.2.patch,
> HADOOP-10561.3.patch, HADOOP-10561.4.patch, HADOOP-10561.5.patch,
> HADOOP-10561.patch
>
>
> The design docs for Xattrs stated that we handle preserve options with copy
> commands
> From doc:
> Preserve option of commands like “cp -p” shell command and “distcp -p” should
> work on XAttrs.
> In the case of source fs supports XAttrs but target fs does not support,
> exception about XAttrs not supported will be thrown.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HADOOP-10681) Compressor inner methods are all synchronized - within a tight loop
[ https://issues.apache.org/jira/browse/HADOOP-10681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HADOOP-10681: Assignee: Gopal V > Compressor inner methods are all synchronized - within a tight loop > --- > > Key: HADOOP-10681 > URL: https://issues.apache.org/jira/browse/HADOOP-10681 > Project: Hadoop Common > Issue Type: Bug > Components: performance >Affects Versions: 2.2.0, 2.4.0, 2.5.0 >Reporter: Gopal V >Assignee: Gopal V > Labels: perfomance > Attachments: compress-cmpxchg-small.png, perf-top-spill-merge.png > > > The current implementation of SnappyCompressor spends more time within the > java loop of copying from the user buffer into the direct buffer allocated to > the compressor impl, than the time it takes to compress the buffers. > !perf-top-spill-merge.png! > The bottleneck was found to be java monitor code inside SnappyCompressor. > The methods are neatly inlined by the JIT into the parent caller > (BlockCompressorStream::write), which unfortunately does not flatten out the > synchronized blocks. > !compress-cmpxchg-small.png! > The loop does a write of small byte[] buffers (each IFile key+value). > I counted approximately 6 monitor enter/exit blocks per k-v pair written. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10681) Remove synchronized blocks from SnappyCodec and ZlibCodec buffering
[ https://issues.apache.org/jira/browse/HADOOP-10681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HADOOP-10681: - Summary: Remove synchronized blocks from SnappyCodec and ZlibCodec buffering (was: Compressor inner methods are all synchronized - within a tight loop) > Remove synchronized blocks from SnappyCodec and ZlibCodec buffering > --- > > Key: HADOOP-10681 > URL: https://issues.apache.org/jira/browse/HADOOP-10681 > Project: Hadoop Common > Issue Type: Bug > Components: performance >Affects Versions: 2.2.0, 2.4.0, 2.5.0 >Reporter: Gopal V >Assignee: Gopal V > Labels: perfomance > Attachments: compress-cmpxchg-small.png, perf-top-spill-merge.png > > > The current implementation of SnappyCompressor spends more time within the > java loop of copying from the user buffer into the direct buffer allocated to > the compressor impl, than the time it takes to compress the buffers. > !perf-top-spill-merge.png! > The bottleneck was found to be java monitor code inside SnappyCompressor. > The methods are neatly inlined by the JIT into the parent caller > (BlockCompressorStream::write), which unfortunately does not flatten out the > synchronized blocks. > !compress-cmpxchg-small.png! > The loop does a write of small byte[] buffers (each IFile key+value). > I counted approximately 6 monitor enter/exit blocks per k-v pair written. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10646) KeyProvider buildVersionName method should be moved to a utils class
[ https://issues.apache.org/jira/browse/HADOOP-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028842#comment-14028842 ] Alejandro Abdelnur commented on HADOOP-10646: - Larry, both could be in the ProviderUtils class > KeyProvider buildVersionName method should be moved to a utils class > > > Key: HADOOP-10646 > URL: https://issues.apache.org/jira/browse/HADOOP-10646 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 3.0.0 > > > The buildVersionName() method should not be part of the KeyProvider public > API because keyversions could be opaque (not built based on the keyname and > key generation counter). > KeyProvider implementations may choose to use buildVersionName() for reasons > such as described in HADOOP-10611. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[
https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HADOOP-10561:
Attachment: HADOOP-10561.5.patch
Add user doc for {{-p}} option of {{fsshell#cp}} command to the patch.
> Copy command with preserve option should handle Xattrs
> --
>
> Key: HADOOP-10561
> URL: https://issues.apache.org/jira/browse/HADOOP-10561
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 3.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Yi Liu
> Attachments: HADOOP-10561.1.patch, HADOOP-10561.2.patch,
> HADOOP-10561.3.patch, HADOOP-10561.4.patch, HADOOP-10561.5.patch,
> HADOOP-10561.patch
>
>
> The design docs for Xattrs stated that we handle preserve options with copy
> commands
> From doc:
> Preserve option of commands like “cp -p” shell command and “distcp -p” should
> work on XAttrs.
> In the case of source fs supports XAttrs but target fs does not support,
> exception about XAttrs not supported will be thrown.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[
https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028788#comment-14028788
]
Yi Liu commented on HADOOP-10561:
-
Thanks [~cnauroth] for nice review.
{quote}
Maybe we just need to update the issue description.
{quote}
Right, I have updated the JIRA description.
{quote}
Can we also please update the shell documentation?
http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/FileSystemShell.html#cp
Currently, it doesn't mention the preserve option at all. We can either roll
the documentation change right into this patch, or file a separate issue for
later follow-up.
{quote}
I will update the user doc for fs shell#cp in the patch of this JIRA later.
> Copy command with preserve option should handle Xattrs
> --
>
> Key: HADOOP-10561
> URL: https://issues.apache.org/jira/browse/HADOOP-10561
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 3.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Yi Liu
> Attachments: HADOOP-10561.1.patch, HADOOP-10561.2.patch,
> HADOOP-10561.3.patch, HADOOP-10561.4.patch, HADOOP-10561.patch
>
>
> The design docs for Xattrs stated that we handle preserve options with copy
> commands
> From doc:
> Preserve option of commands like “cp -p” shell command and “distcp -p” should
> work on XAttrs.
> In the case of source fs supports XAttrs but target fs does not support,
> exception about XAttrs not supported will be thrown.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[ https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10561: Description: The design docs for Xattrs stated that we handle preserve options with copy commands >From doc: Preserve option of commands like “cp -p” shell command and “distcp -p” should work on XAttrs. In the case of source fs supports XAttrs but target fs does not support, exception about XAttrs not supported will be thrown. was: The design docs for Xattrs stated that we handle preserve options with copy commands >From doc: Preserve option of commands like “cp -p” shell command and “distcp -p” should work on XAttrs. In the case of source fs supports XAttrs but target fs does not support, XAttrs will be ignored with warning message > Copy command with preserve option should handle Xattrs > -- > > Key: HADOOP-10561 > URL: https://issues.apache.org/jira/browse/HADOOP-10561 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Yi Liu > Attachments: HADOOP-10561.1.patch, HADOOP-10561.2.patch, > HADOOP-10561.3.patch, HADOOP-10561.4.patch, HADOOP-10561.patch > > > The design docs for Xattrs stated that we handle preserve options with copy > commands > From doc: > Preserve option of commands like “cp -p” shell command and “distcp -p” should > work on XAttrs. > In the case of source fs supports XAttrs but target fs does not support, > exception about XAttrs not supported will be thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10376) Refactor refresh*Protocols into a single generic refreshConfigProtocol
[ https://issues.apache.org/jira/browse/HADOOP-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028751#comment-14028751 ] Chris Li commented on HADOOP-10376: --- Hmm, how about duplicating the functionality, then? So that users can begin to use the new command (which may uncover shortcomings and needed improvements). > Refactor refresh*Protocols into a single generic refreshConfigProtocol > -- > > Key: HADOOP-10376 > URL: https://issues.apache.org/jira/browse/HADOOP-10376 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chris Li >Assignee: Chris Li >Priority: Minor > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10376.patch, HADOOP-10376.patch, > HADOOP-10376.patch, HADOOP-10376.patch, HADOOP-10376.patch, > RefreshFrameworkProposal.pdf > > > See https://issues.apache.org/jira/browse/HADOOP-10285 > There are starting to be too many refresh*Protocols We can refactor them to > use a single protocol with a variable payload to choose what to do. > Thereafter, we can return an indication of success or failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10685) Migrate Standalone Refresh Protocols to the GenericRefreshProto
[ https://issues.apache.org/jira/browse/HADOOP-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028746#comment-14028746 ] Arpit Agarwal commented on HADOOP-10685: Copying my own comment from HADOOP-10376. bq. I don't think there is a immediate need to update the existing refresh calls since we cannot remove them in 2.x anyway for wire compatibility. > Migrate Standalone Refresh Protocols to the GenericRefreshProto > --- > > Key: HADOOP-10685 > URL: https://issues.apache.org/jira/browse/HADOOP-10685 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chris Li >Assignee: Chris Li >Priority: Minor > > Now that we have a GenericRefreshProtocol, we should migrate existing > protocols towards it. > First, we will duplicate the functionality. > If all goes well we can mark the old methods as deprecated, and remove them > later. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10376) Refactor refresh*Protocols into a single generic refreshConfigProtocol
[ https://issues.apache.org/jira/browse/HADOOP-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028740#comment-14028740 ] Arpit Agarwal commented on HADOOP-10376: I don't think there is a immediate need to update the existing refresh calls since we cannot remove them in 2.x anyway for wire compatibility. However it is good to have the generic refresh facility for future use so we don't need to add more protocols. > Refactor refresh*Protocols into a single generic refreshConfigProtocol > -- > > Key: HADOOP-10376 > URL: https://issues.apache.org/jira/browse/HADOOP-10376 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chris Li >Assignee: Chris Li >Priority: Minor > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10376.patch, HADOOP-10376.patch, > HADOOP-10376.patch, HADOOP-10376.patch, HADOOP-10376.patch, > RefreshFrameworkProposal.pdf > > > See https://issues.apache.org/jira/browse/HADOOP-10285 > There are starting to be too many refresh*Protocols We can refactor them to > use a single protocol with a variable payload to choose what to do. > Thereafter, we can return an indication of success or failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10674) Rewrite the PureJavaCrc32 loop for performance improvement
[ https://issues.apache.org/jira/browse/HADOOP-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028743#comment-14028743 ] Gopal V commented on HADOOP-10674: -- [~szetszwo]: Rajesh tested it on the same linux box. Not going to duplicate that. FYI, I hit cross-thread slowdowns when testing out HADOOP-9601. Might be worth writing a threadpool test (for HDFS daemons, not MR tasks). > Rewrite the PureJavaCrc32 loop for performance improvement > -- > > Key: HADOOP-10674 > URL: https://issues.apache.org/jira/browse/HADOOP-10674 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: c10674_20140609.patch, c10674_20140609b.patch, > c10674_20140610.patch > > > Below are some performance improvement opportunities performance improvement > in PureJavaCrc32. > - eliminate "off += 8; len -= 8;" > - replace T8_x_start with hard coded constants > - eliminate c0 - c7 local variables > In my machine, there are 30% to 50% improvement for most of the cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10685) Migrate Standalone Refresh Protocols to the GenericRefreshProto
[ https://issues.apache.org/jira/browse/HADOOP-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10685: -- Description: Now that we have a GenericRefreshProtocol, we should migrate existing protocols towards it. First, we will duplicate the functionality. If all goes well we can mark the old methods as deprecated, and remove them later. > Migrate Standalone Refresh Protocols to the GenericRefreshProto > --- > > Key: HADOOP-10685 > URL: https://issues.apache.org/jira/browse/HADOOP-10685 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chris Li >Assignee: Chris Li >Priority: Minor > > Now that we have a GenericRefreshProtocol, we should migrate existing > protocols towards it. > First, we will duplicate the functionality. > If all goes well we can mark the old methods as deprecated, and remove them > later. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10685) Migrate Standalone Refresh Protocols to the GenericRefreshProto
[ https://issues.apache.org/jira/browse/HADOOP-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10685: -- Summary: Migrate Standalone Refresh Protocols to the GenericRefreshProto (was: Move RefreshCallQueue from its own protocol to the GenericRefreshProto) > Migrate Standalone Refresh Protocols to the GenericRefreshProto > --- > > Key: HADOOP-10685 > URL: https://issues.apache.org/jira/browse/HADOOP-10685 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chris Li >Assignee: Chris Li >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10376) Refactor refresh*Protocols into a single generic refreshConfigProtocol
[ https://issues.apache.org/jira/browse/HADOOP-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028731#comment-14028731 ] Chris Li commented on HADOOP-10376: --- Thanks for reviewing! I think the next step is to start moving other refreshProtos towards the generic proto, marking things as deprecated along the way. I'll start with RefreshCallQueue since it's the newest. HADOOP-10685 to track > Refactor refresh*Protocols into a single generic refreshConfigProtocol > -- > > Key: HADOOP-10376 > URL: https://issues.apache.org/jira/browse/HADOOP-10376 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chris Li >Assignee: Chris Li >Priority: Minor > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10376.patch, HADOOP-10376.patch, > HADOOP-10376.patch, HADOOP-10376.patch, HADOOP-10376.patch, > RefreshFrameworkProposal.pdf > > > See https://issues.apache.org/jira/browse/HADOOP-10285 > There are starting to be too many refresh*Protocols We can refactor them to > use a single protocol with a variable payload to choose what to do. > Thereafter, we can return an indication of success or failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10685) Move RefreshCallQueue from its own protocol to the GenericRefreshProto
Chris Li created HADOOP-10685: - Summary: Move RefreshCallQueue from its own protocol to the GenericRefreshProto Key: HADOOP-10685 URL: https://issues.apache.org/jira/browse/HADOOP-10685 Project: Hadoop Common Issue Type: Improvement Reporter: Chris Li Assignee: Chris Li Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10376) Refactor refresh*Protocols into a single generic refreshConfigProtocol
[ https://issues.apache.org/jira/browse/HADOOP-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-10376: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 for the latest patch. I committed it to trunk and branch-2. Thanks for the contribution Chris! > Refactor refresh*Protocols into a single generic refreshConfigProtocol > -- > > Key: HADOOP-10376 > URL: https://issues.apache.org/jira/browse/HADOOP-10376 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chris Li >Assignee: Chris Li >Priority: Minor > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10376.patch, HADOOP-10376.patch, > HADOOP-10376.patch, HADOOP-10376.patch, HADOOP-10376.patch, > RefreshFrameworkProposal.pdf > > > See https://issues.apache.org/jira/browse/HADOOP-10285 > There are starting to be too many refresh*Protocols We can refactor them to > use a single protocol with a variable payload to choose what to do. > Thereafter, we can return an indication of success or failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10376) Refactor refresh*Protocols into a single generic refreshConfigProtocol
[ https://issues.apache.org/jira/browse/HADOOP-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028703#comment-14028703 ] Hudson commented on HADOOP-10376: - SUCCESS: Integrated in Hadoop-trunk-Commit #5687 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5687/]) HADOOP-10376. Refactor refresh*Protocols into a single generic refreshConfigProtocol. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1602055) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/GenericRefreshProtocol.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RefreshHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RefreshRegistry.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RefreshResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protocolPB/GenericRefreshProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protocolPB/GenericRefreshProtocolPB.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/protocolPB/GenericRefreshProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto/GenericRefreshProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HDFSPolicyProvider.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocols.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/TestGenericRefresh.java > Refactor refresh*Protocols into a single generic refreshConfigProtocol > -- > > Key: HADOOP-10376 > URL: https://issues.apache.org/jira/browse/HADOOP-10376 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chris Li >Assignee: Chris Li >Priority: Minor > Attachments: HADOOP-10376.patch, HADOOP-10376.patch, > HADOOP-10376.patch, HADOOP-10376.patch, HADOOP-10376.patch, > RefreshFrameworkProposal.pdf > > > See https://issues.apache.org/jira/browse/HADOOP-10285 > There are starting to be too many refresh*Protocols We can refactor them to > use a single protocol with a variable payload to choose what to do. > Thereafter, we can return an indication of success or failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10674) Rewrite the PureJavaCrc32 loop for performance improvement
[ https://issues.apache.org/jira/browse/HADOOP-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028688#comment-14028688 ] Tsz Wo Nicholas Sze commented on HADOOP-10674: -- In Java 7 and 8, the new code is faster than the existing code by 10% - 20% in most cases. However, it is still slower than java.util.zip.CRC32. [~rajesh.balamohan], thanks for check Java 7 and 8. You may want to try again with the new patch. [~gopalv], are you going to test the patch? > Rewrite the PureJavaCrc32 loop for performance improvement > -- > > Key: HADOOP-10674 > URL: https://issues.apache.org/jira/browse/HADOOP-10674 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: c10674_20140609.patch, c10674_20140609b.patch, > c10674_20140610.patch > > > Below are some performance improvement opportunities performance improvement > in PureJavaCrc32. > - eliminate "off += 8; len -= 8;" > - replace T8_x_start with hard coded constants > - eliminate c0 - c7 local variables > In my machine, there are 30% to 50% improvement for most of the cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10376) Refactor refresh*Protocols into a single generic refreshConfigProtocol
[
https://issues.apache.org/jira/browse/HADOOP-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028621#comment-14028621
]
Hadoop QA commented on HADOOP-10376:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649882/HADOOP-10376.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4048//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4048//console
This message is automatically generated.
> Refactor refresh*Protocols into a single generic refreshConfigProtocol
> --
>
> Key: HADOOP-10376
> URL: https://issues.apache.org/jira/browse/HADOOP-10376
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chris Li
>Assignee: Chris Li
>Priority: Minor
> Attachments: HADOOP-10376.patch, HADOOP-10376.patch,
> HADOOP-10376.patch, HADOOP-10376.patch, HADOOP-10376.patch,
> RefreshFrameworkProposal.pdf
>
>
> See https://issues.apache.org/jira/browse/HADOOP-10285
> There are starting to be too many refresh*Protocols We can refactor them to
> use a single protocol with a variable payload to choose what to do.
> Thereafter, we can return an indication of success or failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028506#comment-14028506 ] Haohui Mai commented on HADOOP-10389: - bq. So far, we haven't found any defects. So far I'm yet to be convinced that the code is in good quality. I can see two implication on "haven't found any defects". One is the code is in good quality, the other is the code has not yet been gone through full scale reviews and testings. I'm unsure which implication I should get. I haven't gone through this patch, but in other patches there might be some similar issues: https://issues.apache.org/jira/browse/HADOOP-10640?focusedCommentId=14025478&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14025478 Due to the complexity the code, I'm deeply concern that the code cannot be properly reviewed and have at least the same quality of the code in Hadoop. bq. This JIRA doesn't have anything to do with C versus C++. It is about the RPC code, which was already reviewed and committed. You're missing the point. From the prospective of a reviewer, the key question to answer is that what quality the code currently is. There are two problems I can see as the current state of the code: # The code is yet to be separated to logical pieces to be throughly reviewed. # The control flow involves several heap allocations, and it is nontrivial to ensure that there are no resource leaks. I call out C++ because it is widely available when developing native code. You're more than welcome to do it in C if you can address the above requirements. I'm not sure what your plan on this branch, but if you plan to merge it to trunk, in my opinion the above comments have to be properly addressed. bq. And C++ doesn't provide "type checked proofs" of anything, since it is an unsafe language (much like C, which it's based on). If you wanted provable correctness, you might try something like SML or Haskell plus a theorem prover like Coq. But what does this have to do with a native client? At the very least, the code should be free of the following common defects: resource leaks, buffer overflow, and control flow hijacking. It would be great if you can fix your code to systematically show that. I call out C++ because it allows you to provide a much better claim that the code is correct (obviously it can't save you if you intentionally screw it up), but it seems the the current code does not even have this luxury. By the way, if you're interested in how far off C/C++ towards a memory safe language, you might want to take a look at the SAFECode project (http://safecode.cs.illinois.edu/index.html). My impression is that the current code is too far off. bq. If I opened a JIRA about switching Hadoop to Scala, would the discussion continue "in the right tone"? If I pointed out a bunch of stuff that would be "fixed by scala" (that wasn't actually buggy), what would the tone be then? (Note that I have nothing against Scala-- it's a fine language.) There are more interests in writing more robust and efficient code, compared to what language you want to implement it. Personally I'm fairly open to this discussion, if you can demonstrate the benefits as I pointed out in this jira. > Native RPCv9 client > --- > > Key: HADOOP-10389 > URL: https://issues.apache.org/jira/browse/HADOOP-10389 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Binglin Chang >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch, > HADOOP-10389.004.patch, HADOOP-10389.005.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10682) Metrics are not output in trunk
[
https://issues.apache.org/jira/browse/HADOOP-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10682:
---
Description:
Metrics are not output in trunk by the following configuration:
{code}
*.sink.file.class=org.apache.Hadoop.metrics2.sink.FileSink
*.period=10
namenode.sink.file.filename=namenode-metrics.out
{code}
The below change worked well.
{code}
- namenode.sink.file.filename=namenode-metrics.out
+ NameNode.sink.file.filename=namenode-metrics.out
{code}
It means that an old configuration doesn't work on trunk. We should fix it or
document to use "NameNode".
was:
Metrics are not output in trunk by the following configuration:
{code}
*sink.file.class=org.apache.Hadoop.metrics2.sink.FileSink
*.period=10
namenode.sink.file.filename=namenode-metrics.out
{code}
The below change worked well.
{code}
- namenode.sink.file.filename=namenode-metrics.out
+ NameNode.sink.file.filename=namenode-metrics.out
{code}
It means that an old configuration doesn't work on trunk. We should fix it or
document to use "NameNode".
> Metrics are not output in trunk
> ---
>
> Key: HADOOP-10682
> URL: https://issues.apache.org/jira/browse/HADOOP-10682
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Akira AJISAKA
>
> Metrics are not output in trunk by the following configuration:
> {code}
> *.sink.file.class=org.apache.Hadoop.metrics2.sink.FileSink
> *.period=10
> namenode.sink.file.filename=namenode-metrics.out
> {code}
> The below change worked well.
> {code}
> - namenode.sink.file.filename=namenode-metrics.out
> + NameNode.sink.file.filename=namenode-metrics.out
> {code}
> It means that an old configuration doesn't work on trunk. We should fix it or
> document to use "NameNode".
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10622) Shell.runCommand can deadlock
[ https://issues.apache.org/jira/browse/HADOOP-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028480#comment-14028480 ] Hudson commented on HADOOP-10622: - FAILURE: Integrated in Hadoop-trunk-Commit #5685 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5685/]) HADOOP-10622. Shell.runCommand can deadlock. Contributed by Gera Shegalov (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1602033) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java > Shell.runCommand can deadlock > - > > Key: HADOOP-10622 > URL: https://issues.apache.org/jira/browse/HADOOP-10622 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Gera Shegalov >Priority: Critical > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10622.patch, HADOOP-10622.v02.patch > > > Ran into a deadlock in Shell.runCommand. Stacktrace details to follow. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10622) Shell.runCommand can deadlock
[ https://issues.apache.org/jira/browse/HADOOP-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HADOOP-10622: Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Gera! I committed this to trunk and branch-2. > Shell.runCommand can deadlock > - > > Key: HADOOP-10622 > URL: https://issues.apache.org/jira/browse/HADOOP-10622 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Gera Shegalov >Priority: Critical > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10622.patch, HADOOP-10622.v02.patch > > > Ran into a deadlock in Shell.runCommand. Stacktrace details to follow. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028468#comment-14028468 ] Larry McCay commented on HADOOP-10607: -- I don't believe that this metric failure is related to this patch. > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607-10.patch, 10607-11.patch, 10607-2.patch, > 10607-3.patch, 10607-4.patch, 10607-5.patch, 10607-6.patch, 10607-7.patch, > 10607-8.patch, 10607-9.patch, 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10622) Shell.runCommand can deadlock
[ https://issues.apache.org/jira/browse/HADOOP-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028457#comment-14028457 ] Jason Lowe commented on HADOOP-10622: - +1 lgtm. Committing this. > Shell.runCommand can deadlock > - > > Key: HADOOP-10622 > URL: https://issues.apache.org/jira/browse/HADOOP-10622 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Gera Shegalov >Priority: Critical > Attachments: HADOOP-10622.patch, HADOOP-10622.v02.patch > > > Ran into a deadlock in Shell.runCommand. Stacktrace details to follow. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10684) Extend HA support for more use cases
Paul Rubio created HADOOP-10684: --- Summary: Extend HA support for more use cases Key: HADOOP-10684 URL: https://issues.apache.org/jira/browse/HADOOP-10684 Project: Hadoop Common Issue Type: Improvement Components: ha Reporter: Paul Rubio Priority: Minor We'd like the current HA framework to be more configurable from a behavior standpoint. In particular: - Add the ability for a HAServiceTarget to survive a configurable number of health check failures (default of 0) before HealthMonitor (HM) reports service not responding or service unhealthy. For instance, predicate the HM on a state machine whose default implementation can be overridden by method or constructor argument. The default would behave the same as today. -- If a target fails a health check but does not exceed the maximum number of consecutive check failures, it’d be desirable if the target and/or controller were alerted. --- i.e. Introduce a SERVICE_DYING state --Additionally, it’d be desirable if a mechanism existed, similar to fencing semantics, for “reviving” a service that transitioned to SERVICE_DYING. --- i.e. attemptRevive(…) - Add the ability to allow a service to completely fail (no failover or failback possible). There are scenarios where allowing a failover or failback could cause more damage. -- E.g. a recovered master with stale data. The master may have been manually recovered (human error). - Add affinity to a particular HAServiceTarget. -- In other words, allow the controller to prefer one target over another when deciding leadership. -- If a higher affinity, but previously unhealthy target, becomes healthy then it should be allowed to become the leader. -- Likewise, if two targets are racing for a ZooKeeper lock, then the controller should "prefer" the higher the affinity target. -- It might make more sense to add a different implementation/subclass of the ZKFailoverController (i.e. ZKAffinityFailoverController) than modify current behavior. Please comment with thoughts/ideas/etc... Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[
https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028444#comment-14028444
]
Hadoop QA commented on HADOOP-10607:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649866/10607-11.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4047//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4047//console
This message is automatically generated.
> Create an API to Separate Credentials/Password Storage from Applications
>
>
> Key: HADOOP-10607
> URL: https://issues.apache.org/jira/browse/HADOOP-10607
> Project: Hadoop Common
> Issue Type: New Feature
> Components: security
>Reporter: Larry McCay
>Assignee: Larry McCay
> Fix For: 3.0.0
>
> Attachments: 10607-10.patch, 10607-11.patch, 10607-2.patch,
> 10607-3.patch, 10607-4.patch, 10607-5.patch, 10607-6.patch, 10607-7.patch,
> 10607-8.patch, 10607-9.patch, 10607.patch
>
>
> As with the filesystem API, we need to provide a generic mechanism to support
> multiple credential storage mechanisms that are potentially from third
> parties.
> We need the ability to eliminate the storage of passwords and secrets in
> clear text within configuration files or within code.
> Toward that end, I propose an API that is configured using a list of URLs of
> CredentialProviders. The implementation will look for implementations using
> the ServiceLoader interface and thus support third party libraries.
> Two providers will be included in this patch. One using the credentials cache
> in MapReduce jobs and the other using Java KeyStores from either HDFS or
> local file system.
> A CredShell CLI will also be included in this patch which provides the
> ability to manage the credentials within the stores.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10557: --- Description: This issue tracks enhancing FsShell cp to add a new command-line option (-pa) for preserving extended ACLs. (was: This issue tracks enhancing FsShell cp to * preserve extended ACLs by -p option or * add a new command-line option for preserving extended ACLs.) > FsShell -cp -p does not preserve extended ACLs > -- > > Key: HADOOP-10557 > URL: https://issues.apache.org/jira/browse/HADOOP-10557 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, > HADOOP-10557.patch > > > This issue tracks enhancing FsShell cp to add a new command-line option (-pa) > for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028436#comment-14028436 ] Akira AJISAKA commented on HADOOP-10557: Thanks [~cnauroth] for the comments! I wait for HADOOP-10561 to get committed. > FsShell -cp -p does not preserve extended ACLs > -- > > Key: HADOOP-10557 > URL: https://issues.apache.org/jira/browse/HADOOP-10557 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, > HADOOP-10557.patch > > > This issue tracks enhancing FsShell cp to > * preserve extended ACLs by -p option > or > * add a new command-line option for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[
https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028430#comment-14028430
]
Chris Nauroth commented on HADOOP-10557:
Thinking about this a little more, I'm in favor of adding a new command line
option instead of building preservation of ACLs into the existing {{-p}}.
As stated in my last comment, I think the failure behavior is important,
because it prevents accidentally copying without preserving important
authorization rules. For a user copying files across different HDFS clusters,
this would unfortunately create a situation where a file on the source cluster
could not be copied to the destination cluster while using {{-p}}. This has
the potential to break existing processes and scripts if a user suddenly sets
an ACL in the source cluster. When we consider the possibility of a large
recursive {{cp -p}}, the impact of the problem becomes even greater. It could
fail half-way through a large recursive copy and force a lot of manual recovery
work for the operator.
Another consideration is the additional RPC load. This might be unexpected in
very large clusters. Again, the recursive use case magnifies this problem.
I recommend that we introduce a new {{-pa}} option. This way, the new behavior
is opt-in, and existing processes won't encounter the problems described above.
If there is a problem, then the operator can decide if it makes sense to drop
the {{-pa}} argument and rerun the command. They'll still have the capability
to copy and preserve all other attributes. I realize this deviates from some
of the behavior on Unix, but it's a trade-off.
You may want to wait for HADOOP-10561 to get committed before making any
further changes. That patch does something similar for xattrs, and I expect
you'll get the opportunity to reuse some code.
> FsShell -cp -p does not preserve extended ACLs
> --
>
> Key: HADOOP-10557
> URL: https://issues.apache.org/jira/browse/HADOOP-10557
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch,
> HADOOP-10557.patch
>
>
> This issue tracks enhancing FsShell cp to
> * preserve extended ACLs by -p option
> or
> * add a new command-line option for preserving extended ACLs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9648) Fix build native library on mac osx
[ https://issues.apache.org/jira/browse/HADOOP-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028400#comment-14028400 ] Jason Lowe commented on HADOOP-9648: This is primarily a YARN change rather than a change to common. We either should have this JIRA track the hadoop-common change and file a separate for the container-executor fixes or move this to YARN and commit there. Otherwise patch looks good overall, mostly nits: - no check for strdup returning NULL - should have braces even for 1-liners, e.g.: while (*p == '/') - "Failed to mount cgroup controller, not support" should be "Failed to mount cgroup controller, not supported" > Fix build native library on mac osx > --- > > Key: HADOOP-9648 > URL: https://issues.apache.org/jira/browse/HADOOP-9648 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1.0.4, 1.2.0, 1.1.2, 2.0.5-alpha >Reporter: Kirill A. Korinskiy >Assignee: Binglin Chang > Attachments: HADOOP-9648-native-osx.1.0.4.patch, > HADOOP-9648-native-osx.1.1.2.patch, HADOOP-9648-native-osx.1.2.0.patch, > HADOOP-9648-native-osx.2.0.5-alpha-rc1.patch, HADOOP-9648.v2.patch > > > Some patches for fixing build a hadoop native library on os x 10.7/10.8. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10376) Refactor refresh*Protocols into a single generic refreshConfigProtocol
[ https://issues.apache.org/jira/browse/HADOOP-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10376: -- Attachment: HADOOP-10376.patch Ah oops, I accidentally reverted that change with git on my end. Thanks for catching that. `identifier` should now be optional > Refactor refresh*Protocols into a single generic refreshConfigProtocol > -- > > Key: HADOOP-10376 > URL: https://issues.apache.org/jira/browse/HADOOP-10376 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chris Li >Assignee: Chris Li >Priority: Minor > Attachments: HADOOP-10376.patch, HADOOP-10376.patch, > HADOOP-10376.patch, HADOOP-10376.patch, HADOOP-10376.patch, > RefreshFrameworkProposal.pdf > > > See https://issues.apache.org/jira/browse/HADOOP-10285 > There are starting to be too many refresh*Protocols We can refactor them to > use a single protocol with a variable payload to choose what to do. > Thereafter, we can return an indication of success or failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028383#comment-14028383
]
Colin Patrick McCabe commented on HADOOP-10389:
---
bq. Just a random grep, here is a call to vsprintf but not vsnprintf:
The reason why that is a {{vsprintf}} is that we use {{vsnprintf}} above with
the same format string to calculate the length of the buffer we'll need.
bq. Additionally, the caller can easily trigger a buffer overflow by passing
"%s" as a format string. Alternatively, using stringstream will never have this
problem. Note that the code uses the c++ runtime anyway, why not just writing
the code in a type safe way and let the type system do the proof for you?
The caller can't "easily trigger a buffer overflow" in the code above. The
reason is because if the caller passes {{%s}} followed by a string, we will
compute the length of that string as {{fmt_len}}. Let's follow through the
code.
{code}
fmt_len = vsnprintf(test_buf, sizeof(test_buf), fmt, ap);
if (fmt_len < 0) {
ierr = (struct hadoop_err*)&HADOOP_VSNPRINTF_ERR;
goto done;
}
class = errno_to_class(code);
class_len = strlen(class);
msg_len = class_len + SUFFIX_LEN + fmt_len + 1;
err->msg = malloc(msg_len);
if (!err->msg) {
ierr = (struct hadoop_err*)&HADOOP_OOM_ERR;
goto done;
}
strcpy(err->msg, class);
strcpy(err->msg + class_len, SUFFIX);
vsprintf(err->msg + class_len + SUFFIX_LEN, fmt, ap2);
{code}
So you can see that {{msg_len}} will depend on {{fmt_len}}. If the length of
the string the caller supplies to {{%s}} increases, the length of the buffer we
allocate will increase.
bq. valgrind is not a panacea. Valgrind is a dynamic tool where it spots the
leak only when the code runs into the buggy path. Many leaks happen in the
error-handling path, and I'm skeptical that (1) the unit tests are able to
cover all of them, (2) you'll ever be able to run valgrind in production
considering its overheads (10x~100x). Alternatively, the issue is largely
addressed in c++ compile time if the code passes smart pointers with discipline.
C++ is not a panacea. I have worked on large C++ projects in the past. They
have memory leaks too, sometimes quite serious ones. All of them ran valgrind,
thread sanitizer, and other dynamic tools to spot the leaks. Once you get a
complex object ownership structure with {{shared_ptr}}, {{unique_ptr}}, and
{{auto_ptr}}, plus passing bare references and pointers... it's easy to make
mistakes.
bq. In short, they are best-effort approaches to address the common but
critical defects I've raised. With proper uses, a modern language like C++ is
able to provide much higher assurance (type checked proofs) in security and
reliability. Why not embracing these tools instead of putting heavy dependency
and responsibility on code review to catch these bugs?
So far, we haven't found any defects. And C++ doesn't provide "type checked
proofs" of anything, since it is an unsafe language (much like C, which it's
based on), which supports arbitrary casting, pointer aliasing, arbitrary code
execution, and so on. If you wanted provable correctness, you might try
something like {{SML}} or {{Haskell}} plus a theorem prover like {{Coq}}. But
what does this have to do with a native client?
bq. You have already made your point on your c++ cred. Lets continue the
discussion in the right tone.
Suresh, I apologize if this came across as rude. My intention was just to
point out that I have considered the merits of C versus C\+\+. I have
experience with both. It seemed like there was an assumption that I was not
familiar with C\+\+.
I feel somewhat frustrated, because I don't think JIRA is a suitable forum for
programming language advocacy. This JIRA doesn't have anything to do with C
versus C++. It is about the RPC code, which was already reviewed and
committed. There are plenty of places on the web advocating a particular
programming language-- does this have to be one of them? If I opened a JIRA
about switching Hadoop to Scala, would the discussion continue "in the right
tone"? If I pointed out a bunch of stuff that would be "fixed by scala" (that
wasn't actually buggy), what would the tone be then? (Note that I have nothing
against Scala-- it's a fine language.)
> Native RPCv9 client
> ---
>
> Key: HADOOP-10389
> URL: https://issues.apache.org/jira/browse/HADOOP-10389
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Binglin Chang
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch,
> HADOOP-10389.004.patch, HADOOP-10389.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#625
[jira] [Commented] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[
https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028372#comment-14028372
]
Hadoop QA commented on HADOOP-10683:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649863/HADOOP-10683.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4046//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4046//console
This message is automatically generated.
> Users authenticated with KERBEROS are recorded as being authenticated with
> SIMPLE
> -
>
> Key: HADOOP-10683
> URL: https://issues.apache.org/jira/browse/HADOOP-10683
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HADOOP-10683.patch
>
>
> We have enabled kerberos authentication in our clusters, but we see the
> following in the log files
> 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth successful for x...@y.com (*auth:SIMPLE*)
> 2014-06-11 11:07:05,914 INFO
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization successful for x...@y.com (auth:KERBEROS) for
> protocol=interface
> This is quite confusing for administrators.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10607: - Attachment: 10607-11.patch Added missing apache license header. > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607-10.patch, 10607-11.patch, 10607-2.patch, > 10607-3.patch, 10607-4.patch, 10607-5.patch, 10607-6.patch, 10607-7.patch, > 10607-8.patch, 10607-9.patch, 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10607: - Status: Patch Available (was: Open) > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607-10.patch, 10607-11.patch, 10607-2.patch, > 10607-3.patch, 10607-4.patch, 10607-5.patch, 10607-6.patch, 10607-7.patch, > 10607-8.patch, 10607-9.patch, 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10607: - Status: Open (was: Patch Available) > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607-10.patch, 10607-2.patch, 10607-3.patch, > 10607-4.patch, 10607-5.patch, 10607-6.patch, 10607-7.patch, 10607-8.patch, > 10607-9.patch, 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HADOOP-10376) Refactor refresh*Protocols into a single generic refreshConfigProtocol
[
https://issues.apache.org/jira/browse/HADOOP-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028307#comment-14028307
]
Arpit Agarwal edited comment on HADOOP-10376 at 6/11/14 7:49 PM:
-
Hi Chris,
- {{GenericRefreshRequestProto.identifier}} is still required (should be
optional). Looks unintentional since you added a check for {{hasIdentifier}} in
the server side translator.
+1 otherwise. Thanks for addressing the feedback.
was (Author: arpitagarwal):
Hi Chris,
- {{GenericRefreshRequestProto.identifier}} is still required. Looks
unintentional since you added a check for {{hasIdentifier}} in the server side
translator.
+1 otherwise. Thanks for addressing the feedback.
> Refactor refresh*Protocols into a single generic refreshConfigProtocol
> --
>
> Key: HADOOP-10376
> URL: https://issues.apache.org/jira/browse/HADOOP-10376
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chris Li
>Assignee: Chris Li
>Priority: Minor
> Attachments: HADOOP-10376.patch, HADOOP-10376.patch,
> HADOOP-10376.patch, HADOOP-10376.patch, RefreshFrameworkProposal.pdf
>
>
> See https://issues.apache.org/jira/browse/HADOOP-10285
> There are starting to be too many refresh*Protocols We can refactor them to
> use a single protocol with a variable payload to choose what to do.
> Thereafter, we can return an indication of success or failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10376) Refactor refresh*Protocols into a single generic refreshConfigProtocol
[
https://issues.apache.org/jira/browse/HADOOP-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028307#comment-14028307
]
Arpit Agarwal commented on HADOOP-10376:
Hi Chris,
- {{GenericRefreshRequestProto.identifier}} is still required. Looks
unintentional since you added a check for {{hasIdentifier}} in the server side
translator.
+1 otherwise. Thanks for addressing the feedback.
> Refactor refresh*Protocols into a single generic refreshConfigProtocol
> --
>
> Key: HADOOP-10376
> URL: https://issues.apache.org/jira/browse/HADOOP-10376
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chris Li
>Assignee: Chris Li
>Priority: Minor
> Attachments: HADOOP-10376.patch, HADOOP-10376.patch,
> HADOOP-10376.patch, HADOOP-10376.patch, RefreshFrameworkProposal.pdf
>
>
> See https://issues.apache.org/jira/browse/HADOOP-10285
> There are starting to be too many refresh*Protocols We can refactor them to
> use a single protocol with a variable payload to choose what to do.
> Thereafter, we can return an indication of success or failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[
https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028302#comment-14028302
]
Hadoop QA commented on HADOOP-10607:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649841/10607-10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4045//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4045//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4045//console
This message is automatically generated.
> Create an API to Separate Credentials/Password Storage from Applications
>
>
> Key: HADOOP-10607
> URL: https://issues.apache.org/jira/browse/HADOOP-10607
> Project: Hadoop Common
> Issue Type: New Feature
> Components: security
>Reporter: Larry McCay
>Assignee: Larry McCay
> Fix For: 3.0.0
>
> Attachments: 10607-10.patch, 10607-2.patch, 10607-3.patch,
> 10607-4.patch, 10607-5.patch, 10607-6.patch, 10607-7.patch, 10607-8.patch,
> 10607-9.patch, 10607.patch
>
>
> As with the filesystem API, we need to provide a generic mechanism to support
> multiple credential storage mechanisms that are potentially from third
> parties.
> We need the ability to eliminate the storage of passwords and secrets in
> clear text within configuration files or within code.
> Toward that end, I propose an API that is configured using a list of URLs of
> CredentialProviders. The implementation will look for implementations using
> the ServiceLoader interface and thus support third party libraries.
> Two providers will be included in this patch. One using the credentials cache
> in MapReduce jobs and the other using Java KeyStores from either HDFS or
> local file system.
> A CredShell CLI will also be included in this patch which provides the
> ability to manage the credentials within the stores.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[ https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10683: -- Status: Patch Available (was: Open) > Users authenticated with KERBEROS are recorded as being authenticated with > SIMPLE > - > > Key: HADOOP-10683 > URL: https://issues.apache.org/jira/browse/HADOOP-10683 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10683.patch > > > We have enabled kerberos authentication in our clusters, but we see the > following in the log files > 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for x...@y.com (*auth:SIMPLE*) > 2014-06-11 11:07:05,914 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for x...@y.com (auth:KERBEROS) for > protocol=interface > This is quite confusing for administrators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[ https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10683: -- Attachment: HADOOP-10683.patch Attaching a patch which overloads _UserGroupInformation.createRemoteUser_ to accept AuthMethod as an additional parameter. The current _UserGroupInformation.createRemoteUser_ is modified to invoke overloaded method with AuthMethod.SIMPLE. A test case is also added. > Users authenticated with KERBEROS are recorded as being authenticated with > SIMPLE > - > > Key: HADOOP-10683 > URL: https://issues.apache.org/jira/browse/HADOOP-10683 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10683.patch > > > We have enabled kerberos authentication in our clusters, but we see the > following in the log files > 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for x...@y.com (*auth:SIMPLE*) > 2014-06-11 11:07:05,914 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for x...@y.com (auth:KERBEROS) for > protocol=interface > This is quite confusing for administrators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[
https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028294#comment-14028294
]
Benoy Antony commented on HADOOP-10683:
---
This is because AuthenticationMethod is hardcoded to SIMPLE in
_UserGroupInformation.createRemoteUser_
{code:title=UserGroupInformation.createRemoteUser|borderStyle=solid}
UserGroupInformation result = new UserGroupInformation(subject);
result.setAuthenticationMethod(AuthenticationMethod.SIMPLE);
{code}
> Users authenticated with KERBEROS are recorded as being authenticated with
> SIMPLE
> -
>
> Key: HADOOP-10683
> URL: https://issues.apache.org/jira/browse/HADOOP-10683
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Benoy Antony
>Assignee: Benoy Antony
>
> We have enabled kerberos authentication in our clusters, but we see the
> following in the log files
> 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth successful for x...@y.com (*auth:SIMPLE*)
> 2014-06-11 11:07:05,914 INFO
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization successful for x...@y.com (auth:KERBEROS) for
> protocol=interface
> This is quite confusing for administrators.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
Benoy Antony created HADOOP-10683: - Summary: Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE Key: HADOOP-10683 URL: https://issues.apache.org/jira/browse/HADOOP-10683 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Benoy Antony Assignee: Benoy Antony We have enabled kerberos authentication in our clusters, but we see the following in the log files 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for x...@y.com (*auth:SIMPLE*) 2014-06-11 11:07:05,914 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for x...@y.com (auth:KERBEROS) for protocol=interface This is quite confusing for administrators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10656) The password keystore file is not picked by LDAP group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028249#comment-14028249
]
Hudson commented on HADOOP-10656:
-
SUCCESS: Integrated in Hadoop-trunk-Commit #5683 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5683/])
HADOOP-10656. The password keystore file is not picked by LDAP group mapping.
Contributed by Brandon Li (brandonli:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1601985)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java
> The password keystore file is not picked by LDAP group mapping
> --
>
> Key: HADOOP-10656
> URL: https://issues.apache.org/jira/browse/HADOOP-10656
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Fix For: 2.5.0
>
> Attachments: HADOOP-10656.002.patch, HADOOP-10656.003.patch,
> HADOOP-10656.patch
>
>
> The user configured password file(LDAP_KEYSTORE_PASSWORD_FILE_KEY) will not
> be picked by LdapGroupsMapping:
> In setConf():
> {noformat}
> keystorePass =
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY, LDAP_KEYSTORE_PASSWORD_DEFAULT);
> if (keystorePass.isEmpty()) {
> keystorePass = extractPassword(
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY,
> LDAP_KEYSTORE_PASSWORD_DEFAULT));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[
https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028229#comment-14028229
]
Chris Nauroth commented on HADOOP-10557:
[~ajisakaa], on Unix, it's true that {{cp -p}} attempts to preserve the ACL,
but it's possible for this to fail if the destination is on a different file
system that doesn't support ACLs. When that happens, {{cp}} doesn't just log a
warning. It fails with a non-zero exit code. See below for a transcript
demonstrating this on Ubuntu. My system is set up so that /home/cnauroth is on
a file system with ACL support, but /mnt/share is on a file system without ACL
support.
I think failure is desirable in this case. A user might be surprised that a
sensitive file was opened for access after successfully copying it, and log
messages don't always give sufficient visibility into the problem.
{code}
[root@ubuntu:pts/0] ~
> touch /home/cnauroth/fileWithAcl
[root@ubuntu:pts/0] ~
> setfacl -m user:cnauroth:rwx /home/cnauroth/fileWithAcl
[root@ubuntu:pts/0] ~
> cp /home/cnauroth/fileWithAcl /mnt/share/cnauroth/fileWithAclNoPreserve
[root@ubuntu:pts/0] ~
> cp -p /home/cnauroth/fileWithAcl /mnt/share/cnauroth/fileWithAclPreserve
cp: preserving permissions for `/mnt/share/cnauroth/fileWithAclPreserve':
Operation not supported
[root@ubuntu:pts/0] ~
> echo $?
1
{code}
> FsShell -cp -p does not preserve extended ACLs
> --
>
> Key: HADOOP-10557
> URL: https://issues.apache.org/jira/browse/HADOOP-10557
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch,
> HADOOP-10557.patch
>
>
> This issue tracks enhancing FsShell cp to
> * preserve extended ACLs by -p option
> or
> * add a new command-line option for preserving extended ACLs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10656) The password keystore file is not picked by LDAP group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028228#comment-14028228
]
Brandon Li commented on HADOOP-10656:
-
Thank you, Chris, for the review. I've committed the patch.
> The password keystore file is not picked by LDAP group mapping
> --
>
> Key: HADOOP-10656
> URL: https://issues.apache.org/jira/browse/HADOOP-10656
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Fix For: 2.5.0
>
> Attachments: HADOOP-10656.002.patch, HADOOP-10656.003.patch,
> HADOOP-10656.patch
>
>
> The user configured password file(LDAP_KEYSTORE_PASSWORD_FILE_KEY) will not
> be picked by LdapGroupsMapping:
> In setConf():
> {noformat}
> keystorePass =
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY, LDAP_KEYSTORE_PASSWORD_DEFAULT);
> if (keystorePass.isEmpty()) {
> keystorePass = extractPassword(
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY,
> LDAP_KEYSTORE_PASSWORD_DEFAULT));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10656) The password keystore file is not picked by LDAP group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HADOOP-10656:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> The password keystore file is not picked by LDAP group mapping
> --
>
> Key: HADOOP-10656
> URL: https://issues.apache.org/jira/browse/HADOOP-10656
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Fix For: 2.5.0
>
> Attachments: HADOOP-10656.002.patch, HADOOP-10656.003.patch,
> HADOOP-10656.patch
>
>
> The user configured password file(LDAP_KEYSTORE_PASSWORD_FILE_KEY) will not
> be picked by LdapGroupsMapping:
> In setConf():
> {noformat}
> keystorePass =
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY, LDAP_KEYSTORE_PASSWORD_DEFAULT);
> if (keystorePass.isEmpty()) {
> keystorePass = extractPassword(
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY,
> LDAP_KEYSTORE_PASSWORD_DEFAULT));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10656) The password keystore file is not picked by LDAP group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HADOOP-10656:
Fix Version/s: 2.5.0
> The password keystore file is not picked by LDAP group mapping
> --
>
> Key: HADOOP-10656
> URL: https://issues.apache.org/jira/browse/HADOOP-10656
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Fix For: 2.5.0
>
> Attachments: HADOOP-10656.002.patch, HADOOP-10656.003.patch,
> HADOOP-10656.patch
>
>
> The user configured password file(LDAP_KEYSTORE_PASSWORD_FILE_KEY) will not
> be picked by LdapGroupsMapping:
> In setConf():
> {noformat}
> keystorePass =
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY, LDAP_KEYSTORE_PASSWORD_DEFAULT);
> if (keystorePass.isEmpty()) {
> keystorePass = extractPassword(
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY,
> LDAP_KEYSTORE_PASSWORD_DEFAULT));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10682) Metrics are not output in trunk
Akira AJISAKA created HADOOP-10682:
--
Summary: Metrics are not output in trunk
Key: HADOOP-10682
URL: https://issues.apache.org/jira/browse/HADOOP-10682
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Reporter: Akira AJISAKA
Metrics are not output in trunk by the following configuration:
{code}
*sink.file.class=org.apache.Hadoop.metrics2.sink.FileSink
*.period=10
namenode.sink.file.filename=namenode-metrics.out
{code}
The below change worked well.
{code}
- namenode.sink.file.filename=namenode-metrics.out
+ NameNode.sink.file.filename=namenode-metrics.out
{code}
It means that an old configuration doesn't work on trunk. We should fix it or
document to use "NameNode".
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028221#comment-14028221 ] Suresh Srinivas commented on HADOOP-10389: -- [~wheat9], can you please provide information on how many lines of code is not necessary and can be replaced by c++ standard libraries. Of course that is one of the factors and not the only reason to choose the direction of this solution. > Native RPCv9 client > --- > > Key: HADOOP-10389 > URL: https://issues.apache.org/jira/browse/HADOOP-10389 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Binglin Chang >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch, > HADOOP-10389.004.patch, HADOOP-10389.005.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10607: - Status: Patch Available (was: Open) > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607-10.patch, 10607-2.patch, 10607-3.patch, > 10607-4.patch, 10607-5.patch, 10607-6.patch, 10607-7.patch, 10607-8.patch, > 10607-9.patch, 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028201#comment-14028201 ] Suresh Srinivas commented on HADOOP-10389: -- Can you please tone down the rudeness. The following comments are unnecessary for productive discussion: bq. I'm very familiar with C++ and I don't need a lecture on its advantages, having been a user for a decade. You have already made your point on your c++ cred. Lets continue the discussion in the right tone. > Native RPCv9 client > --- > > Key: HADOOP-10389 > URL: https://issues.apache.org/jira/browse/HADOOP-10389 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Binglin Chang >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch, > HADOOP-10389.004.patch, HADOOP-10389.005.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10656) The password keystore file is not picked by LDAP group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028155#comment-14028155
]
Hadoop QA commented on HADOOP-10656:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649833/HADOOP-10656.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4044//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4044//console
This message is automatically generated.
> The password keystore file is not picked by LDAP group mapping
> --
>
> Key: HADOOP-10656
> URL: https://issues.apache.org/jira/browse/HADOOP-10656
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HADOOP-10656.002.patch, HADOOP-10656.003.patch,
> HADOOP-10656.patch
>
>
> The user configured password file(LDAP_KEYSTORE_PASSWORD_FILE_KEY) will not
> be picked by LdapGroupsMapping:
> In setConf():
> {noformat}
> keystorePass =
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY, LDAP_KEYSTORE_PASSWORD_DEFAULT);
> if (keystorePass.isEmpty()) {
> keystorePass = extractPassword(
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY,
> LDAP_KEYSTORE_PASSWORD_DEFAULT));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[
https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028146#comment-14028146
]
Chris Nauroth commented on HADOOP-10561:
I have one more question. The issue description says this:
bq. In the case of source fs supports XAttrs but target fs does not support,
XAttrs will be ignored with warning message
It looks like the patch instead causes the command to fail if the target
doesn't support xattrs. It calls {{FileSystem#setXAttr}}, and it lets
exceptions propagate out. I think this behavior is preferable, especially with
consideration that xattrs are used in the implementation of encryption zones.
If the user requested preserving xattrs, then dropping xattrs related to
encryption could accidentally open access to encrypted data at the target, and
a log message wouldn't give enough visibility into the problem.
Maybe we just need to update the issue description.
> Copy command with preserve option should handle Xattrs
> --
>
> Key: HADOOP-10561
> URL: https://issues.apache.org/jira/browse/HADOOP-10561
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 3.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Yi Liu
> Attachments: HADOOP-10561.1.patch, HADOOP-10561.2.patch,
> HADOOP-10561.3.patch, HADOOP-10561.4.patch, HADOOP-10561.patch
>
>
> The design docs for Xattrs stated that we handle preserve options with copy
> commands
> From doc:
> Preserve option of commands like “cp -p” shell command and “distcp -p” should
> work on XAttrs.
> In the case of source fs supports XAttrs but target fs does not support,
> XAttrs will be ignored
> with warning message
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028136#comment-14028136
]
Haohui Mai commented on HADOOP-10389:
-
bq. printf and similar functions are not a problem because we have
_attribute_((format(printf))), which warns about cases where the format string
doesn't match the varargs. And we only ever use snprintf, vsnprintf, and the
other functions which print into a buffer of a known size.
Just a random grep, here is a call to {{vsprintf}} but not {{vsnprintf}}:
{code}
+vsprintf(err->msg + class_len + SUFFIX_LEN, fmt, ap2);
+va_end(ap2);
+va_end(ap);
{code}
While trying to enforce good coding style is a good thing, it is a best-effort
claim but not a proof. I'm not trying to question your creditability on the
claim, as any human can make mistakes on such a large code base.
The point is the type checker in compiler gives you a proof automatically. With
proper uses, compilers do this job much better than humans, so please leave the
job to the compiler.
Additionally, the caller can easily trigger a buffer overflow by passing "%s"
as a format string. Alternatively, using {{stringstream}} will never have this
problem. Note that the code uses the c++ runtime anyway, why not just writing
the code in a type safe way and let the type system do the proof for you?
bq. I use valgrind to spot memory leaks.
valgrind is not a panacea. Valgrind is a dynamic tool where it spots the leak
only when the code runs into the buggy path. Many leaks happen in the
error-handling path, and I'm skeptical that (1) the unit tests are able to
cover all of them, (2) you'll ever be able to run valgrind in production
considering its overheads (10x~100x). Alternatively, the issue is largely
addressed in c++ compile time if the code passes smart pointers with discipline.
In short, they are best-effort approaches to address the common but critical
defects I've raised. With proper uses, a modern language like C++ is able to
provide much higher assurance (type checked proofs) in security and
reliability. Why not embracing these tools instead of putting heavy dependency
and responsibility on code review to catch these bugs?
> Native RPCv9 client
> ---
>
> Key: HADOOP-10389
> URL: https://issues.apache.org/jira/browse/HADOOP-10389
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Binglin Chang
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch,
> HADOOP-10389.004.patch, HADOOP-10389.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10607: - Status: Open (was: Patch Available) > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607-10.patch, 10607-2.patch, 10607-3.patch, > 10607-4.patch, 10607-5.patch, 10607-6.patch, 10607-7.patch, 10607-8.patch, > 10607-9.patch, 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10646) KeyProvider buildVersionName method should be moved to a utils class
[ https://issues.apache.org/jira/browse/HADOOP-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028110#comment-14028110 ] Larry McCay commented on HADOOP-10646: -- Hi [~tucu00] - After looking at it more closely, I'm not sure that the methods should be in the same utils class. BuildVersion is pretty much key provider specific where unnestUri is used across both Key and CredentialProvider APIs. I added unnestUri to a new org.apache.hadoop.security.ProviderUtils class - since it is more general security provider functionality. Thanks. > KeyProvider buildVersionName method should be moved to a utils class > > > Key: HADOOP-10646 > URL: https://issues.apache.org/jira/browse/HADOOP-10646 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 3.0.0 > > > The buildVersionName() method should not be part of the KeyProvider public > API because keyversions could be opaque (not built based on the keyname and > key generation counter). > KeyProvider implementations may choose to use buildVersionName() for reasons > such as described in HADOOP-10611. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10681) Compressor inner methods are all synchronized - within a tight loop
Gopal V created HADOOP-10681: Summary: Compressor inner methods are all synchronized - within a tight loop Key: HADOOP-10681 URL: https://issues.apache.org/jira/browse/HADOOP-10681 Project: Hadoop Common Issue Type: Bug Components: performance Affects Versions: 2.4.0, 2.2.0, 2.5.0 Reporter: Gopal V Attachments: compress-cmpxchg-small.png, perf-top-spill-merge.png The current implementation of SnappyCompressor spends more time within the java loop of copying from the user buffer into the direct buffer allocated to the compressor impl, than the time it takes to compress the buffers. !perf-top-spill-merge.png! The bottleneck was found to be java monitor code inside SnappyCompressor. The methods are neatly inlined by the JIT into the parent caller (BlockCompressorStream::write), which unfortunately does not flatten out the synchronized blocks. !compress-cmpxchg-small.png! The loop does a write of small byte[] buffers (each IFile key+value). I counted approximately 6 monitor enter/exit blocks per k-v pair written. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10607: - Attachment: 10607-10.patch Addresses the points in [~owen.omalley]'s review. > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607-10.patch, 10607-2.patch, 10607-3.patch, > 10607-4.patch, 10607-5.patch, 10607-6.patch, 10607-7.patch, 10607-8.patch, > 10607-9.patch, 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10681) Compressor inner methods are all synchronized - within a tight loop
[ https://issues.apache.org/jira/browse/HADOOP-10681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HADOOP-10681: - Attachment: perf-top-spill-merge.png > Compressor inner methods are all synchronized - within a tight loop > --- > > Key: HADOOP-10681 > URL: https://issues.apache.org/jira/browse/HADOOP-10681 > Project: Hadoop Common > Issue Type: Bug > Components: performance >Affects Versions: 2.2.0, 2.4.0, 2.5.0 >Reporter: Gopal V > Labels: perfomance > Attachments: compress-cmpxchg-small.png, perf-top-spill-merge.png > > > The current implementation of SnappyCompressor spends more time within the > java loop of copying from the user buffer into the direct buffer allocated to > the compressor impl, than the time it takes to compress the buffers. > !perf-top-spill-merge.png! > The bottleneck was found to be java monitor code inside SnappyCompressor. > The methods are neatly inlined by the JIT into the parent caller > (BlockCompressorStream::write), which unfortunately does not flatten out the > synchronized blocks. > !compress-cmpxchg-small.png! > The loop does a write of small byte[] buffers (each IFile key+value). > I counted approximately 6 monitor enter/exit blocks per k-v pair written. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10681) Compressor inner methods are all synchronized - within a tight loop
[ https://issues.apache.org/jira/browse/HADOOP-10681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HADOOP-10681: - Attachment: compress-cmpxchg-small.png > Compressor inner methods are all synchronized - within a tight loop > --- > > Key: HADOOP-10681 > URL: https://issues.apache.org/jira/browse/HADOOP-10681 > Project: Hadoop Common > Issue Type: Bug > Components: performance >Affects Versions: 2.2.0, 2.4.0, 2.5.0 >Reporter: Gopal V > Labels: perfomance > Attachments: compress-cmpxchg-small.png, perf-top-spill-merge.png > > > The current implementation of SnappyCompressor spends more time within the > java loop of copying from the user buffer into the direct buffer allocated to > the compressor impl, than the time it takes to compress the buffers. > !perf-top-spill-merge.png! > The bottleneck was found to be java monitor code inside SnappyCompressor. > The methods are neatly inlined by the JIT into the parent caller > (BlockCompressorStream::write), which unfortunately does not flatten out the > synchronized blocks. > !compress-cmpxchg-small.png! > The loop does a write of small byte[] buffers (each IFile key+value). > I counted approximately 6 monitor enter/exit blocks per k-v pair written. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10680) Document metrics included in MXBean
Akira AJISAKA created HADOOP-10680:
--
Summary: Document metrics included in MXBean
Key: HADOOP-10680
URL: https://issues.apache.org/jira/browse/HADOOP-10680
Project: Hadoop Common
Issue Type: Improvement
Components: documentation
Reporter: Akira AJISAKA
Enhancement of HADOOP-6350.
For example, SnapshotInfo metrics in NameNode, DataNodeInfo and FsDatasetState
metrics in DataNode are not collected by {{MetricsSystem}} but registered to
{{MXBean}} via {{MBeanServer}}, so these metrics can be seen by jmx/jconsole.
These metrics should also be documented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10656) The password keystore file is not picked by LDAP group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028076#comment-14028076
]
Chris Nauroth commented on HADOOP-10656:
+1 for the v3 patch, pending Jenkins run. Thanks again, Brandon!
> The password keystore file is not picked by LDAP group mapping
> --
>
> Key: HADOOP-10656
> URL: https://issues.apache.org/jira/browse/HADOOP-10656
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HADOOP-10656.002.patch, HADOOP-10656.003.patch,
> HADOOP-10656.patch
>
>
> The user configured password file(LDAP_KEYSTORE_PASSWORD_FILE_KEY) will not
> be picked by LdapGroupsMapping:
> In setConf():
> {noformat}
> keystorePass =
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY, LDAP_KEYSTORE_PASSWORD_DEFAULT);
> if (keystorePass.isEmpty()) {
> keystorePass = extractPassword(
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY,
> LDAP_KEYSTORE_PASSWORD_DEFAULT));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[ https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028074#comment-14028074 ] Chris Nauroth commented on HADOOP-10561: The patch looks good, but can we also please update the shell documentation? http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/FileSystemShell.html#cp Currently, it doesn't mention the preserve option at all. We can either roll the documentation change right into this patch, or file a separate issue for later follow-up. > Copy command with preserve option should handle Xattrs > -- > > Key: HADOOP-10561 > URL: https://issues.apache.org/jira/browse/HADOOP-10561 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Yi Liu > Attachments: HADOOP-10561.1.patch, HADOOP-10561.2.patch, > HADOOP-10561.3.patch, HADOOP-10561.4.patch, HADOOP-10561.patch > > > The design docs for Xattrs stated that we handle preserve options with copy > commands > From doc: > Preserve option of commands like “cp -p” shell command and “distcp -p” should > work on XAttrs. > In the case of source fs supports XAttrs but target fs does not support, > XAttrs will be ignored > with warning message -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028072#comment-14028072
]
Colin Patrick McCabe commented on HADOOP-10389:
---
bq. I'm also seeing many calls to strcpy(), as well as calls to *printf() with
non constant format strings.
There's 10 calls to {{strcpy}} in the whole code base, which is 37 files now.
All of them are in cases where we calculate the destination buffer size
beforehand based on the source string size, so there is no problem.
{{printf}} and similar functions are not a problem because we have
{{__attribute__((format(printf)))}}, which warns about cases where the format
string doesn't match the varargs. And we only ever use {{snprintf}},
{{vsnprintf}}, and the other functions which print into a buffer of a known
size.
bq. My question is that (1) whether the code contains no memory leak, no buffer
overflow, and no format string overflow? (2) whether the code always passes the
function pointer with the correct type? I'm perfectly happy to +1 your patches
as long as you can show your code is indeed free of these common defects.
The compiler checks whether function pointers are the correct type. We have
compile-time checks that printf's arguments match its format string, we don't
ever use non-constant format strings, and we use the versions that take a
maximum length . I use {{valgrind}} to spot memory leaks.
I think running static and dynamic analysis tools on the code is always a good
idea. Having good unit tests coverage is also a good idea. The native code
will always have burdens that Java doesn't have, because it is not garbage
collected. But I think with care, those burdens can be managed in a client,
just like we manage them in the existing libhdfs.
> Native RPCv9 client
> ---
>
> Key: HADOOP-10389
> URL: https://issues.apache.org/jira/browse/HADOOP-10389
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Binglin Chang
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch,
> HADOOP-10389.004.patch, HADOOP-10389.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10656) The password keystore file is not picked by LDAP group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028060#comment-14028060
]
Brandon Li commented on HADOOP-10656:
-
Thank you, Chris. IOUtils#cleanup makes more sense and the change is smaller.
Updated the patch.
> The password keystore file is not picked by LDAP group mapping
> --
>
> Key: HADOOP-10656
> URL: https://issues.apache.org/jira/browse/HADOOP-10656
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HADOOP-10656.002.patch, HADOOP-10656.003.patch,
> HADOOP-10656.patch
>
>
> The user configured password file(LDAP_KEYSTORE_PASSWORD_FILE_KEY) will not
> be picked by LdapGroupsMapping:
> In setConf():
> {noformat}
> keystorePass =
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY, LDAP_KEYSTORE_PASSWORD_DEFAULT);
> if (keystorePass.isEmpty()) {
> keystorePass = extractPassword(
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY,
> LDAP_KEYSTORE_PASSWORD_DEFAULT));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10656) The password keystore file is not picked by LDAP group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HADOOP-10656:
Attachment: HADOOP-10656.003.patch
> The password keystore file is not picked by LDAP group mapping
> --
>
> Key: HADOOP-10656
> URL: https://issues.apache.org/jira/browse/HADOOP-10656
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HADOOP-10656.002.patch, HADOOP-10656.003.patch,
> HADOOP-10656.patch
>
>
> The user configured password file(LDAP_KEYSTORE_PASSWORD_FILE_KEY) will not
> be picked by LdapGroupsMapping:
> In setConf():
> {noformat}
> keystorePass =
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY, LDAP_KEYSTORE_PASSWORD_DEFAULT);
> if (keystorePass.isEmpty()) {
> keystorePass = extractPassword(
> conf.get(LDAP_KEYSTORE_PASSWORD_KEY,
> LDAP_KEYSTORE_PASSWORD_DEFAULT));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10663: --- Assignee: Remus Rusanu > Path.getFileSystem should identify Windows drive (c:) like files and return > the localFS > --- > > Key: HADOOP-10663 > URL: https://issues.apache.org/jira/browse/HADOOP-10663 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Remus Rusanu >Assignee: Remus Rusanu > Attachments: HADOOP-10663.1.patch > > > While investigating the problem with MAPREDUCE-5912 I considered that the > fact that asking a Windows specific Path to get its file system should not > return the default FS, but the localFS. Specifically, I believe that > Path.initialize should identify such names as belonging to the 'file://' > scheme (when null) and so create appropriate URIs. > I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027864#comment-14027864 ] Hudson commented on HADOOP-9629: FAILURE: Integrated in Hadoop-Mapreduce-trunk #1798 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1798/]) HADOOP-9629. Support Windows Azure Storage - Blob as a file system in Hadoop. Contributed by Dexter Bradshaw, Mostafa Elhemali, Xi Fang, Johannes Klein, David Lao, Mike Liddell, Chuan Liu, Lengning Liu, Ivan Mitic, Michael Rys, Alexander Stojanovic, Brian Swan, and Min Wei. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1601781) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure * /hadoop/common/trunk/hadoop-tools/hadoop-azure/.gitignore * /hadoop/common/trunk/hadoop-tools/hadoop-azure/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-azure/dev-support * /hadoop/common/trunk/hadoop-tools/hadoop-azure/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/config * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/config/checkstyle.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureException.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlobMaterialization.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/FileMetadata.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/KeyProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/KeyProviderException.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeFileSystemStore.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PartialListing.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfThrottlingIntercept.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SendRequestIntercept.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/ShellDecryptionKeyProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SimpleKeyProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/Wasb.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbFsck.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/package.html * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/InMemoryBlockBlobStore.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java * /hadoop/common/trunk/hadoo
[jira] [Commented] (HADOOP-9555) HA functionality that uses ZooKeeper may experience inadvertent TCP RST and miss session expiration event due to bug in client connection management
[ https://issues.apache.org/jira/browse/HADOOP-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027866#comment-14027866 ] Hudson commented on HADOOP-9555: FAILURE: Integrated in Hadoop-Mapreduce-trunk #1798 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1798/]) HADOOP-9555. HA functionality that uses ZooKeeper may experience inadvertent TCP RST and miss session expiration event due to bug in client connection management. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1601709) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-project/pom.xml > HA functionality that uses ZooKeeper may experience inadvertent TCP RST and > miss session expiration event due to bug in client connection management > > > Key: HADOOP-9555 > URL: https://issues.apache.org/jira/browse/HADOOP-9555 > Project: Hadoop Common > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-9555.1.patch > > > ZOOKEEPER-1702 tracks a client connection management bug. The bug can cause > an unexpected TCP RST that ultimately prevents delivery of a session > expiration event. The symptoms of the bug seem to show up more frequently on > Windows than on other platforms (though it's not really a Windows-specific > bug). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027786#comment-14027786 ] Hudson commented on HADOOP-9629: SUCCESS: Integrated in Hadoop-Hdfs-trunk #1771 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1771/]) HADOOP-9629. Support Windows Azure Storage - Blob as a file system in Hadoop. Contributed by Dexter Bradshaw, Mostafa Elhemali, Xi Fang, Johannes Klein, David Lao, Mike Liddell, Chuan Liu, Lengning Liu, Ivan Mitic, Michael Rys, Alexander Stojanovic, Brian Swan, and Min Wei. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1601781) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure * /hadoop/common/trunk/hadoop-tools/hadoop-azure/.gitignore * /hadoop/common/trunk/hadoop-tools/hadoop-azure/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-azure/dev-support * /hadoop/common/trunk/hadoop-tools/hadoop-azure/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/config * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/config/checkstyle.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureException.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlobMaterialization.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/FileMetadata.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/KeyProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/KeyProviderException.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeFileSystemStore.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PartialListing.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfThrottlingIntercept.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SendRequestIntercept.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/ShellDecryptionKeyProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SimpleKeyProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/Wasb.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbFsck.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/package.html * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/InMemoryBlockBlobStore.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java * /hadoop/common/trunk/hadoop-tools/ha
[jira] [Commented] (HADOOP-9555) HA functionality that uses ZooKeeper may experience inadvertent TCP RST and miss session expiration event due to bug in client connection management
[ https://issues.apache.org/jira/browse/HADOOP-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027788#comment-14027788 ] Hudson commented on HADOOP-9555: SUCCESS: Integrated in Hadoop-Hdfs-trunk #1771 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1771/]) HADOOP-9555. HA functionality that uses ZooKeeper may experience inadvertent TCP RST and miss session expiration event due to bug in client connection management. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1601709) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-project/pom.xml > HA functionality that uses ZooKeeper may experience inadvertent TCP RST and > miss session expiration event due to bug in client connection management > > > Key: HADOOP-9555 > URL: https://issues.apache.org/jira/browse/HADOOP-9555 > Project: Hadoop Common > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-9555.1.patch > > > ZOOKEEPER-1702 tracks a client connection management bug. The bug can cause > an unexpected TCP RST that ultimately prevents delivery of a session > expiration event. The symptoms of the bug seem to show up more frequently on > Windows than on other platforms (though it's not really a Windows-specific > bug). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027638#comment-14027638 ] Hudson commented on HADOOP-9629: FAILURE: Integrated in Hadoop-Yarn-trunk #580 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/580/]) HADOOP-9629. Support Windows Azure Storage - Blob as a file system in Hadoop. Contributed by Dexter Bradshaw, Mostafa Elhemali, Xi Fang, Johannes Klein, David Lao, Mike Liddell, Chuan Liu, Lengning Liu, Ivan Mitic, Michael Rys, Alexander Stojanovic, Brian Swan, and Min Wei. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1601781) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure * /hadoop/common/trunk/hadoop-tools/hadoop-azure/.gitignore * /hadoop/common/trunk/hadoop-tools/hadoop-azure/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-azure/dev-support * /hadoop/common/trunk/hadoop-tools/hadoop-azure/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/config * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/config/checkstyle.xml * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureException.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlobMaterialization.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/FileMetadata.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/KeyProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/KeyProviderException.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeFileSystemStore.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PartialListing.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfThrottlingIntercept.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SendRequestIntercept.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/ShellDecryptionKeyProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SimpleKeyProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/Wasb.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbFsck.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/package.html * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/InMemoryBlockBlobStore.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java * /hadoop/common/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java * /hadoop/common/trunk/hadoop-tools/hado
[jira] [Commented] (HADOOP-9555) HA functionality that uses ZooKeeper may experience inadvertent TCP RST and miss session expiration event due to bug in client connection management
[ https://issues.apache.org/jira/browse/HADOOP-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027640#comment-14027640 ] Hudson commented on HADOOP-9555: FAILURE: Integrated in Hadoop-Yarn-trunk #580 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/580/]) HADOOP-9555. HA functionality that uses ZooKeeper may experience inadvertent TCP RST and miss session expiration event due to bug in client connection management. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1601709) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-project/pom.xml > HA functionality that uses ZooKeeper may experience inadvertent TCP RST and > miss session expiration event due to bug in client connection management > > > Key: HADOOP-9555 > URL: https://issues.apache.org/jira/browse/HADOOP-9555 > Project: Hadoop Common > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-9555.1.patch > > > ZOOKEEPER-1702 tracks a client connection management bug. The bug can cause > an unexpected TCP RST that ultimately prevents delivery of a session > expiration event. The symptoms of the bug seem to show up more frequently on > Windows than on other platforms (though it's not really a Windows-specific > bug). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10641) Introduce Coordination Engine
[ https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027601#comment-14027601 ] Rakesh R commented on HADOOP-10641: --- bq.The intent of this jira is not to solve the general problem of distributed consensus. That is, I do not propose to build an implementation of paxos or other coordination algorithms here. This is only to introduce a common interface, so that real implementations such as ZooKeeper could be plugged into hadoop projects. This sounds interesting. Thanks for the effort!. If I understood the discussion correctly here, the idea is to build a quorum based replication. For example, the events(I think this represents data) are submitted as proposals to a quorum of nodes. In ZooKeeper terms, Leader proposes values to the Followers. Now Leader wait for acknowledgements from a quorum of Followers before considering a proposal committed. Also, Leader queues COMMIT(zxid) events to all Followers so that all other nodes learn the events. This ensures that the events will be reached to all nodes in the system. Adding one more point, in general ZK provides strong ordering guarantees. Sometime back ZooKeeper folks initiated discussions to decouple ZAB from ZooKeeper, so that users can make use of this and can define their own models and reliably replicate the data. There is a related JIRA ZOOKEEPER-1931 talks similar feature, now this is in initial dev stage. Please have a look at this. I hope this would help to define a common interface, also an opportunity for us to know more about the use cases. Regards, Rakesh > Introduce Coordination Engine > - > > Key: HADOOP-10641 > URL: https://issues.apache.org/jira/browse/HADOOP-10641 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Konstantin Shvachko >Assignee: Plamen Jeliazkov > Attachments: HADOOP-10641.patch, HADOOP-10641.patch, > HADOOP-10641.patch > > > Coordination Engine (CE) is a system, which allows to agree on a sequence of > events in a distributed system. In order to be reliable CE should be > distributed by itself. > Coordination Engine can be based on different algorithms (paxos, raft, 2PC, > zab) and have different implementations, depending on use cases, reliability, > availability, and performance requirements. > CE should have a common API, so that it could serve as a pluggable component > in different projects. The immediate beneficiaries are HDFS (HDFS-6469) and > HBase (HBASE-10909). > First implementation is proposed to be based on ZooKeeper. -- This message was sent by Atlassian JIRA (v6.2#6252)
