[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[
https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881707#comment-13881707
]
Hadoop QA commented on HDFS-5804:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12625100/HDFS-5804.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5940//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5940//console
This message is automatically generated.
> HDFS NFS Gateway fails to mount and proxy when using Kerberos
> -
>
> Key: HDFS-5804
> URL: https://issues.apache.org/jira/browse/HDFS-5804
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Abin Shahab
> Attachments: HDFS-5804.patch, HDFS-5804.patch, HDFS-5804.patch,
> exception-as-root.log, javadoc-after-patch.log, javadoc-before-patch.log
>
>
> When using HDFS nfs gateway with secure hadoop
> (hadoop.security.authentication: kerberos), mounting hdfs fails.
> Additionally, there is no mechanism to support proxy user(nfs needs to proxy
> as the user invoking commands on the hdfs mount).
> Steps to reproduce:
> 1) start a hadoop cluster with kerberos enabled.
> 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has
> a an account in kerberos.
> 3) Get the keytab for nfsserver, and issue the following mount command: mount
> -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point
> 4) You'll see in the nfsserver logs that Kerberos is complaining about not
> having a TGT for root.
> This is the stacktrace:
> java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is:
> "my-nfs-server-host.com/10.252.4.197"; destination host is:
> "my-namenode-host.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
> at org.apache.hadoop.ipc.Client.call(Client.java:1351)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664)
> at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891)
> at
> org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUps
[jira] [Commented] (HDFS-5728) [Diskfull] Block recovery will fail if the metafile does not have crc for all chunks of the block
[ https://issues.apache.org/jira/browse/HDFS-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881686#comment-13881686 ] Hudson commented on HDFS-5728: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5036 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5036/]) HDFS-5728. Block recovery will fail if the metafile does not have crc for all chunks of the block. Contributed by Vinay. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1561223) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecovery.java > [Diskfull] Block recovery will fail if the metafile does not have crc for all > chunks of the block > - > > Key: HDFS-5728 > URL: https://issues.apache.org/jira/browse/HDFS-5728 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.10, 2.2.0 >Reporter: Vinay >Assignee: Vinay >Priority: Critical > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5728.patch, HDFS-5728.patch, HDFS-5728.patch > > > 1. Client (regionsever) has opened stream to write its WAL to HDFS. This is > not one time upload, data will be written slowly. > 2. One of the DataNode got diskfull ( due to some other data filled up disks) > 3. Unfortunately block was being written to only this datanode in cluster, so > client write has also failed. > 4. After some time disk is made free and all processes are restarted. > 5. Now HMaster try to recover the file by calling recoverLease. > At this time recovery was failing saying file length mismatch. > When checked, > actual block file length: 62484480 > Calculated block length: 62455808 > This was because, metafile was having crc for only 62455808 bytes, and it > considered 62455808 as the block size. > No matter how many times, recovery was continously failing. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5241) Provide alternate queuing audit logger to reduce logging contention
[ https://issues.apache.org/jira/browse/HDFS-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881684#comment-13881684 ] Hudson commented on HDFS-5241: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5036 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5036/]) HDFS-5241. Provide alternate queuing audit logger to reduce logging contention (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560761) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java > Provide alternate queuing audit logger to reduce logging contention > --- > > Key: HDFS-5241 > URL: https://issues.apache.org/jira/browse/HDFS-5241 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-5241.patch, HDFS-5241.patch > > > The default audit logger has extremely poor performance. The internal > synchronization of log4j causes massive contention between the call handlers > (100 by default) which drastically limits the throughput of the NN. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5788) listLocatedStatus response can be very large
[ https://issues.apache.org/jira/browse/HDFS-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881679#comment-13881679 ] Hudson commented on HDFS-5788: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5036 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5036/]) HDFS-5788. listLocatedStatus response can be very large. Contributed by Nathan Roberts. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560750) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java > listLocatedStatus response can be very large > > > Key: HDFS-5788 > URL: https://issues.apache.org/jira/browse/HDFS-5788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Nathan Roberts >Assignee: Nathan Roberts > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5788.patch > > > Currently we limit the size of listStatus requests to a default of 1000 > entries. This works fine except in the case of listLocatedStatus where the > location information can be quite large. As an example, a directory with 7000 > entries, 4 blocks each, 3 way replication - a listLocatedStatus response is > over 1MB. This can chew up very large amounts of memory in the NN if lots of > clients try to do this simultaneously. > Seems like it would be better if we also considered the amount of location > information being returned when deciding how many files to return. > Patch will follow shortly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5789) Some of snapshot APIs missing checkOperation double check in fsn
[ https://issues.apache.org/jira/browse/HDFS-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881678#comment-13881678 ] Hudson commented on HDFS-5789: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5036 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5036/]) HDFS-5789. Some of snapshot APIs missing checkOperation double check in fsn. Contributed by Uma Maheswara Rao G. (umamahesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1560731) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Some of snapshot APIs missing checkOperation double check in fsn > > > Key: HDFS-5789 > URL: https://issues.apache.org/jira/browse/HDFS-5789 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 3.0.0, 2.3.0 > > Attachments: HDFS-5789.patch > > > HDFS-4591 introduced double checked for HA state while taking fsn lock. > checkoperation made before actually taking lock and after the lock again. > This pattern missed in some of the snapshot APIs and cache management related > APIs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881661#comment-13881661 ] Colin Patrick McCabe commented on HDFS-5776: I might be misunderstanding, but it seems like this should be a client setting, not a datanode setting. Right? > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776-v8.txt, > HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5702) FsShell Cli: Add XML based End-to-End test for getfacl and setfacl commands
[ https://issues.apache.org/jira/browse/HDFS-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881572#comment-13881572 ] Chris Nauroth commented on HDFS-5702: - Vinay, in addition to the list above, could you please also add a fourth test? 4. setfacl -x mask:: This test will fail until HADOOP-10277 gets fixed. I have a patch in progress now. > FsShell Cli: Add XML based End-to-End test for getfacl and setfacl commands > --- > > Key: HDFS-5702 > URL: https://issues.apache.org/jira/browse/HDFS-5702 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode, security >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5702.patch > > > FsShell Cli: Add XML based End-to-End test for getfacl and setfacl commands -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5631) Expose interfaces required by FsDatasetSpi implementations
[ https://issues.apache.org/jira/browse/HDFS-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Powell updated HDFS-5631: --- Attachment: HDFS-5631.patch > Expose interfaces required by FsDatasetSpi implementations > -- > > Key: HDFS-5631 > URL: https://issues.apache.org/jira/browse/HDFS-5631 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 3.0.0 >Reporter: David Powell >Priority: Minor > Attachments: HDFS-5631.patch, HDFS-5631.patch > > > This sub-task addresses section 4.1 of the document attached to HDFS-5194, > the exposure of interfaces needed by a FsDatasetSpi implementation. > Specifically it makes ChunkChecksum public and BlockMetadataHeader's > readHeader() and writeHeader() methods public. > The changes to BlockReaderUtil (and related classes) discussed by section > 4.1 are only needed if supporting short-circuit, and should be addressed > as part of an effort to provide such support rather than this JIRA. > To help ensure these changes are complete and are not regressed in the > future, tests that gauge the accessibility (though *not* behavior) > of interfaces needed by a FsDatasetSpi subclass are also included. > These take the form of a dummy FsDatasetSpi subclass -- a successful > compilation is effectively a pass. Trivial unit tests are included so > that there is something tangible to track. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5631) Expose interfaces required by FsDatasetSpi implementations
[ https://issues.apache.org/jira/browse/HDFS-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Powell updated HDFS-5631: --- Target Version/s: 3.0.0 Status: Patch Available (was: Open) > Expose interfaces required by FsDatasetSpi implementations > -- > > Key: HDFS-5631 > URL: https://issues.apache.org/jira/browse/HDFS-5631 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 3.0.0 >Reporter: David Powell >Priority: Minor > Attachments: HDFS-5631.patch, HDFS-5631.patch > > > This sub-task addresses section 4.1 of the document attached to HDFS-5194, > the exposure of interfaces needed by a FsDatasetSpi implementation. > Specifically it makes ChunkChecksum public and BlockMetadataHeader's > readHeader() and writeHeader() methods public. > The changes to BlockReaderUtil (and related classes) discussed by section > 4.1 are only needed if supporting short-circuit, and should be addressed > as part of an effort to provide such support rather than this JIRA. > To help ensure these changes are complete and are not regressed in the > future, tests that gauge the accessibility (though *not* behavior) > of interfaces needed by a FsDatasetSpi subclass are also included. > These take the form of a dummy FsDatasetSpi subclass -- a successful > compilation is effectively a pass. Trivial unit tests are included so > that there is something tangible to track. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5698) Use protobuf to serialize / deserialize FSImage
[ https://issues.apache.org/jira/browse/HDFS-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881543#comment-13881543 ] Kihwal Lee commented on HDFS-5698: -- What will be the performance/size implications of this? > Use protobuf to serialize / deserialize FSImage > --- > > Key: HDFS-5698 > URL: https://issues.apache.org/jira/browse/HDFS-5698 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5698.000.patch > > > Currently, the code serializes FSImage using in-house serialization > mechanisms. There are a couple disadvantages of the current approach: > # Mixing the responsibility of reconstruction and serialization / > deserialization. The current code paths of serialization / deserialization > have spent a lot of effort on maintaining compatibility. What is worse is > that they are mixed with the complex logic of reconstructing the namespace, > making the code difficult to follow. > # Poor documentation of the current FSImage format. The format of the FSImage > is practically defined by the implementation. An bug in implementation means > a bug in the specification. Furthermore, it also makes writing third-party > tools quite difficult. > # Changing schemas is non-trivial. Adding a field in FSImage requires bumping > the layout version every time. Bumping out layout version requires (1) the > users to explicitly upgrade the clusters, and (2) putting new code to > maintain backward compatibility. > This jira proposes to use protobuf to serialize the FSImage. Protobuf has > been used to serialize / deserialize the RPC message in Hadoop. > Protobuf addresses all the above problems. It clearly separates the > responsibility of serialization and reconstructing the namespace. The > protobuf files document the current format of the FSImage. The developers now > can add optional fields with ease, since the old code can always read the new > FSImage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5830) WebHdfsFileSystem.getFileBlockLocations throws IllegalArgumentException when accessing another cluster.
[
https://issues.apache.org/jira/browse/HDFS-5830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881535#comment-13881535
]
Kihwal Lee commented on HDFS-5830:
--
Marking it as a blocker since WebHdfsFileSystem should remain compatible.
{{toLocatedBlock()}} should use a different version of constructor if cached
location is null.
> WebHdfsFileSystem.getFileBlockLocations throws IllegalArgumentException when
> accessing another cluster.
>
>
> Key: HDFS-5830
> URL: https://issues.apache.org/jira/browse/HDFS-5830
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: caching, hdfs-client
>Affects Versions: 2.3.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Blocker
>
> WebHdfsFileSystem.getFileBlockLocations throws IllegalArgumentException when
> accessing a another cluster (that doesn't have caching support).
> java.lang.IllegalArgumentException: cachedLocs should not be null, use a
> different constructor
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.hdfs.protocol.LocatedBlock.(LocatedBlock.java:79)
> at org.apache.hadoop.hdfs.web.JsonUtil.toLocatedBlock(JsonUtil.java:414)
> at org.apache.hadoop.hdfs.web.JsonUtil.toLocatedBlockList(JsonUtil.java:446)
> at org.apache.hadoop.hdfs.web.JsonUtil.toLocatedBlocks(JsonUtil.java:479)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileBlockLocations(WebHdfsFileSystem.java:1067)
> at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1812)
> at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1797)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5830) WebHdfsFileSystem.getFileBlockLocations throws IllegalArgumentException when accessing another cluster.
[ https://issues.apache.org/jira/browse/HDFS-5830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5830: - Priority: Blocker (was: Major) Target Version/s: 2.3.0 > WebHdfsFileSystem.getFileBlockLocations throws IllegalArgumentException when > accessing another cluster. > > > Key: HDFS-5830 > URL: https://issues.apache.org/jira/browse/HDFS-5830 > Project: Hadoop HDFS > Issue Type: Bug > Components: caching, hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Blocker > > WebHdfsFileSystem.getFileBlockLocations throws IllegalArgumentException when > accessing a another cluster (that doesn't have caching support). > java.lang.IllegalArgumentException: cachedLocs should not be null, use a > different constructor > at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) > at org.apache.hadoop.hdfs.protocol.LocatedBlock.(LocatedBlock.java:79) > at org.apache.hadoop.hdfs.web.JsonUtil.toLocatedBlock(JsonUtil.java:414) > at org.apache.hadoop.hdfs.web.JsonUtil.toLocatedBlockList(JsonUtil.java:446) > at org.apache.hadoop.hdfs.web.JsonUtil.toLocatedBlocks(JsonUtil.java:479) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileBlockLocations(WebHdfsFileSystem.java:1067) > at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1812) > at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1797) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5728) [Diskfull] Block recovery will fail if the metafile does not have crc for all chunks of the block
[ https://issues.apache.org/jira/browse/HDFS-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881518#comment-13881518 ] Kihwal Lee commented on HDFS-5728: -- I've committed this to trunk and branch-2. Thank you for reporting and working on the patch, Vinay. Thanks for the reveiw, Uma. > [Diskfull] Block recovery will fail if the metafile does not have crc for all > chunks of the block > - > > Key: HDFS-5728 > URL: https://issues.apache.org/jira/browse/HDFS-5728 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.10, 2.2.0 >Reporter: Vinay >Assignee: Vinay >Priority: Critical > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5728.patch, HDFS-5728.patch, HDFS-5728.patch > > > 1. Client (regionsever) has opened stream to write its WAL to HDFS. This is > not one time upload, data will be written slowly. > 2. One of the DataNode got diskfull ( due to some other data filled up disks) > 3. Unfortunately block was being written to only this datanode in cluster, so > client write has also failed. > 4. After some time disk is made free and all processes are restarted. > 5. Now HMaster try to recover the file by calling recoverLease. > At this time recovery was failing saying file length mismatch. > When checked, > actual block file length: 62484480 > Calculated block length: 62455808 > This was because, metafile was having crc for only 62455808 bytes, and it > considered 62455808 as the block size. > No matter how many times, recovery was continously failing. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5728) [Diskfull] Block recovery will fail if the metafile does not have crc for all chunks of the block
[ https://issues.apache.org/jira/browse/HDFS-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5728: - Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > [Diskfull] Block recovery will fail if the metafile does not have crc for all > chunks of the block > - > > Key: HDFS-5728 > URL: https://issues.apache.org/jira/browse/HDFS-5728 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.10, 2.2.0 >Reporter: Vinay >Assignee: Vinay >Priority: Critical > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5728.patch, HDFS-5728.patch, HDFS-5728.patch > > > 1. Client (regionsever) has opened stream to write its WAL to HDFS. This is > not one time upload, data will be written slowly. > 2. One of the DataNode got diskfull ( due to some other data filled up disks) > 3. Unfortunately block was being written to only this datanode in cluster, so > client write has also failed. > 4. After some time disk is made free and all processes are restarted. > 5. Now HMaster try to recover the file by calling recoverLease. > At this time recovery was failing saying file length mismatch. > When checked, > actual block file length: 62484480 > Calculated block length: 62455808 > This was because, metafile was having crc for only 62455808 bytes, and it > considered 62455808 as the block size. > No matter how many times, recovery was continously failing. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5728) [Diskfull] Block recovery will fail if the metafile does not have crc for all chunks of the block
[ https://issues.apache.org/jira/browse/HDFS-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5728: - Summary: [Diskfull] Block recovery will fail if the metafile does not have crc for all chunks of the block (was: [Diskfull] Block recovery will fail if the metafile not having crc for all chunks of the block) > [Diskfull] Block recovery will fail if the metafile does not have crc for all > chunks of the block > - > > Key: HDFS-5728 > URL: https://issues.apache.org/jira/browse/HDFS-5728 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.10, 2.2.0 >Reporter: Vinay >Assignee: Vinay >Priority: Critical > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5728.patch, HDFS-5728.patch, HDFS-5728.patch > > > 1. Client (regionsever) has opened stream to write its WAL to HDFS. This is > not one time upload, data will be written slowly. > 2. One of the DataNode got diskfull ( due to some other data filled up disks) > 3. Unfortunately block was being written to only this datanode in cluster, so > client write has also failed. > 4. After some time disk is made free and all processes are restarted. > 5. Now HMaster try to recover the file by calling recoverLease. > At this time recovery was failing saying file length mismatch. > When checked, > actual block file length: 62484480 > Calculated block length: 62455808 > This was because, metafile was having crc for only 62455808 bytes, and it > considered 62455808 as the block size. > No matter how many times, recovery was continously failing. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5728) [Diskfull] Block recovery will fail if the metafile not having crc for all chunks of the block
[ https://issues.apache.org/jira/browse/HDFS-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5728: - Target Version/s: 2.4.0 (was: 2.4.0, 0.23.11) > [Diskfull] Block recovery will fail if the metafile not having crc for all > chunks of the block > -- > > Key: HDFS-5728 > URL: https://issues.apache.org/jira/browse/HDFS-5728 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.10, 2.2.0 >Reporter: Vinay >Assignee: Vinay >Priority: Critical > Attachments: HDFS-5728.patch, HDFS-5728.patch, HDFS-5728.patch > > > 1. Client (regionsever) has opened stream to write its WAL to HDFS. This is > not one time upload, data will be written slowly. > 2. One of the DataNode got diskfull ( due to some other data filled up disks) > 3. Unfortunately block was being written to only this datanode in cluster, so > client write has also failed. > 4. After some time disk is made free and all processes are restarted. > 5. Now HMaster try to recover the file by calling recoverLease. > At this time recovery was failing saying file length mismatch. > When checked, > actual block file length: 62484480 > Calculated block length: 62455808 > This was because, metafile was having crc for only 62455808 bytes, and it > considered 62455808 as the block size. > No matter how many times, recovery was continously failing. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5728) [Diskfull] Block recovery will fail if the metafile not having crc for all chunks of the block
[ https://issues.apache.org/jira/browse/HDFS-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881506#comment-13881506 ] Kihwal Lee commented on HDFS-5728: -- The build server has been down for more than a day, so precommit won't run any time soon. +1 I won't wait for the build server to return. The previous version of patch was fine except the lines removed in the latest version. > [Diskfull] Block recovery will fail if the metafile not having crc for all > chunks of the block > -- > > Key: HDFS-5728 > URL: https://issues.apache.org/jira/browse/HDFS-5728 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.10, 2.2.0 >Reporter: Vinay >Assignee: Vinay >Priority: Critical > Attachments: HDFS-5728.patch, HDFS-5728.patch, HDFS-5728.patch > > > 1. Client (regionsever) has opened stream to write its WAL to HDFS. This is > not one time upload, data will be written slowly. > 2. One of the DataNode got diskfull ( due to some other data filled up disks) > 3. Unfortunately block was being written to only this datanode in cluster, so > client write has also failed. > 4. After some time disk is made free and all processes are restarted. > 5. Now HMaster try to recover the file by calling recoverLease. > At this time recovery was failing saying file length mismatch. > When checked, > actual block file length: 62484480 > Calculated block length: 62455808 > This was because, metafile was having crc for only 62455808 bytes, and it > considered 62455808 as the block size. > No matter how many times, recovery was continously failing. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5620) NameNode: implement Global ACL Set as a memory optimization.
[
https://issues.apache.org/jira/browse/HDFS-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5620:
Attachment: aclRamSizingEstimates.png
aclRamSizingEstimates
> NameNode: implement Global ACL Set as a memory optimization.
>
>
> Key: HDFS-5620
> URL: https://issues.apache.org/jira/browse/HDFS-5620
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS ACLs (HDFS-4685)
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5620.1.patch, aclRamSizingEstimates,
> aclRamSizingEstimates.png
>
>
> The {{AclManager}} can maintain a Global ACL Set to store all distinct ACLs
> in use by the file system. All inodes that have the same ACL entries can
> share the same ACL instance.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Resolved] (HDFS-5620) NameNode: implement Global ACL Set as a memory optimization.
[
https://issues.apache.org/jira/browse/HDFS-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-5620.
-
Resolution: Won't Fix
I'm resolving this issue as won't fix. [~wheat9] was right to question whether
or not we really need this optimization.
Now that ACLs implementation is almost done, I can get a more accurate
measurement of the memory impact using jmap. I'm including a plot that
estimates memory consumption by ACLs in a presumed 50 GB NameNode, trying to
store up to 100 million inodes. I plotted 3 different usage scenarios:
# max usage: All inodes have an ACL, and each one has the maximum ACL entries.
# mid usage: Half the inodes have an ACL, and each ACL has 10 entries.
# low usage: 10% of inodes have an ACL, and each ACL has 2 entries.
For each scenario, I plotted the unoptimized memory consumption and also the
optimized memory consumption, assuming that only 20% of the ACLs are unique.
Each scenario uses a different line color. The unoptimized version uses a
solid line, and the optimized version uses a dashed line. I'm also attaching
the GnuPlot script I wrote to generate this.
The plot demonstrates that ACL usage really needs to get quite high (very large
number of inodes with a very high proportion of them having ACLs) before this
memory optimization starts to provide a benefit.
I am +1 for skipping this for now. We can always resurrect this patch if we
observe a need for it based on real world usage of ACLs. I'm also going to put
this information into a new revision of the design doc on HDFS-4685.
BTW, if we do resurrect this patch, then I probably wouldn't commit the exact
HDFS-5620.1.patch that I put together quickly as a prototype. The Guava
interner has a spinlock inside of it, which we don't really need, because this
would only be accessed under the namesystem write lock anyway.
> NameNode: implement Global ACL Set as a memory optimization.
>
>
> Key: HDFS-5620
> URL: https://issues.apache.org/jira/browse/HDFS-5620
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS ACLs (HDFS-4685)
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5620.1.patch, aclRamSizingEstimates,
> aclRamSizingEstimates.png
>
>
> The {{AclManager}} can maintain a Global ACL Set to store all distinct ACLs
> in use by the file system. All inodes that have the same ACL entries can
> share the same ACL instance.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5356) MiniDFSCluster shoud close all open FileSystems when shutdown()
[ https://issues.apache.org/jira/browse/HDFS-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5356: - Target Version/s: 2.4.0 (was: 2.3.0) > MiniDFSCluster shoud close all open FileSystems when shutdown() > --- > > Key: HDFS-5356 > URL: https://issues.apache.org/jira/browse/HDFS-5356 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.2.0 >Reporter: haosdent >Priority: Critical > Attachments: HDFS-5356.patch > > > After add some metrics functions to DFSClient, I found that some unit tests > relates to metrics are failed. Because MiniDFSCluster are never close open > FileSystems, DFSClients are alive after MiniDFSCluster shutdown(). The > metrics of DFSClients in DefaultMetricsSystem are still exist and this make > other unit tests failed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881375#comment-13881375 ] Abin Shahab commented on HDFS-5804: --- Jing, let me know if you have any feedback on my patch. > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, HDFS-5804.patch, > exception-as-root.log, javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channe
[jira] [Updated] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abin Shahab updated HDFS-5804: -- Attachment: HDFS-5804.patch Updated documentation and param names > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, HDFS-5804.patch, > exception-as-root.log, javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(Defa
[jira] [Updated] (HDFS-5796) The file system browser in the namenode UI requires SPNEGO.
[ https://issues.apache.org/jira/browse/HDFS-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5796: - Target Version/s: 2.4.0 > The file system browser in the namenode UI requires SPNEGO. > --- > > Key: HDFS-5796 > URL: https://issues.apache.org/jira/browse/HDFS-5796 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Priority: Critical > > After HDFS-5382, the browser makes webhdfs REST calls directly, requiring > SPNEGO to work between user's browser and namenode. This won't work if the > cluster's security infrastructure is isolated from the regular network. > Moreover, SPNEGO is not supposed to be required for user-facing web pages. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881341#comment-13881341
]
Arpit Agarwal commented on HDFS-5776:
-
I reviewed the v8 patch. The implementation of {{hedgedFetchBlockByteRange}}
looks great. Nice use of synchronization tools to make the code easy to
understand.
{quote}
how much? 5000? 50? or sth else? i think if enabling this feature
explicitly, the end user/application should know a little backgroud at least,
right?
since we don't have any knowledge about end-user's storage configuration, just
image if they have a fast flash(with HDFS-2832 enabled), say fusionio, probably
one real disk read only cost tens of microseconds,
{quote}
[~xieliang007] #2 and #4 from my previous comment remain unaddressed.
Threads are not free. If you really want to provide a user configurable setting
for the thread count there should be a limit on the order of 64/128. I leave
the exact number to you. The best approach is to use a small multiple of the
processor count.
If an app is not well behaved then the absence of limits can create a positive
feedback loop. The slower the storage layer the more threads will get created
when the correct behavior under load should be back off. Please add a thread
count limit or ideally let’s not expose this setting at all.
The same goes for the delay. Please add a lower bound. The exact value is up to
you. We can always revisit the value if it turns out to be a bottleneck.
{quote}
let's keep it there, it's more easier to understand for developer or end user.
{quote}
I don't think it helps to have these functions and as Jing pointed out there is
no purpose for it. I think it would be best to leave a single config setting
i.e. either a boolean or a thread count, and a single method
{{#isHedgedReadsEnabled}} to query the status of the feature.
{quote}
yes, but the loop is very very light, only when some exceptions like
AccessControlException/InvalidEncryptionKeyException/InvalidBlockTokenException
happened, will do extra loop, and all those have a fast quit mechanism, like
refetchToken/refetchEncryptionKey or disableLegacyBlockReaderLocal, so this
loop will only be executed just a very few times
...
yes, you had a misunderstanding here. that's why i catch IOException fbae
around fetchBlockAt. If we don't catch here, there will be always new refetch
from outside loop and will have a spin loop
{quote}
I still do not understand how you are guarding against multiple refetch.
Previously these counters were initialized outside any loop, now they are being
reinitialized inside a loop.
{{chooseDataNode(LocatedBlock block)}} function looks redundant and should be
removed.
> Support 'hedged' reads in DFSClient
> ---
>
> Key: HDFS-5776
> URL: https://issues.apache.org/jira/browse/HDFS-5776
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt,
> HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776-v8.txt,
> HDFS-5776.txt
>
>
> This is a placeholder of hdfs related stuff backport from
> https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" &
> "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read
> ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we
> could export the interested metric valus into client system(e.g. HBase's
> regionserver metric).
> The core logic is in pread code path, we decide to goto the original
> fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per
> the above config items.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5709) Improve upgrade with existing files and directories named ".snapshot"
[ https://issues.apache.org/jira/browse/HDFS-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881336#comment-13881336 ] Andrew Wang commented on HDFS-5709: --- I'm not sure if this adds anything to the discussion, but the current workflow in the patch is: * By default, the config option is not set * On upgrade, if we hit a .snapshot path, we throw an exception and abort telling the user to set the config option * When the config option is set, there are log prints whenever something is renamed This feels procedurally pretty similar to the alternative of a special startup option, and it's only required to be set if we hit a .snapshot path. Most users will never need to worry about this config. I agree that doing renames with the context of the full parent path (e.g. /X/.snapshot renames to /X/.myx-snapshot, /Y/.snapshot renames to /Y/.myy.snapshot) would be more powerful, but IIUC that implies scanning the fsimage and logs initially to enumerate all the .snapshot paths, have the user configure their rename rules, then load the fsimage/logs again to do the rename. Suresh, does any of this influence your view? I'd really like to move forward on this issue. > Improve upgrade with existing files and directories named ".snapshot" > - > > Key: HDFS-5709 > URL: https://issues.apache.org/jira/browse/HDFS-5709 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: snapshots, upgrade > Attachments: hdfs-5709-1.patch, hdfs-5709-2.patch, hdfs-5709-3.patch, > hdfs-5709-4.patch, hdfs-5709-5.patch > > > Right now in trunk, upgrade fails messily if the old fsimage or edits refer > to a directory named ".snapshot". We should at least print a better error > message (which I believe was the original intention in HDFS-4666), and [~atm] > proposed automatically renaming these files and directories. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Moved] (HDFS-5831) TestAuditLogs#testAuditAllowedStat sometimes fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu moved HBASE-10415 to HDFS-5831:
--
Key: HDFS-5831 (was: HBASE-10415)
Project: Hadoop HDFS (was: HBase)
> TestAuditLogs#testAuditAllowedStat sometimes fails in trunk
> ---
>
> Key: HDFS-5831
> URL: https://issues.apache.org/jira/browse/HDFS-5831
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Ted Yu
>
> Running TestAuditLogs on Linux, I got:
> {code}
> testAuditAllowedStat[1](org.apache.hadoop.hdfs.server.namenode.TestAuditLogs)
> Time elapsed: 6.677 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:92)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertNotNull(Assert.java:526)
> at org.junit.Assert.assertNotNull(Assert.java:537)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.verifyAuditLogsRepeat(TestAuditLogs.java:312)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.verifyAuditLogs(TestAuditLogs.java:295)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.testAuditAllowedStat(TestAuditLogs.java:163)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5831) TestAuditLogs#testAuditAllowedStat sometimes fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HDFS-5831:
-
Attachment:
5831-org.apache.hadoop.hdfs.server.namenode.TestAuditLogs-output.txt
Test output.
> TestAuditLogs#testAuditAllowedStat sometimes fails in trunk
> ---
>
> Key: HDFS-5831
> URL: https://issues.apache.org/jira/browse/HDFS-5831
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Ted Yu
> Attachments:
> 5831-org.apache.hadoop.hdfs.server.namenode.TestAuditLogs-output.txt
>
>
> Running TestAuditLogs on Linux, I got:
> {code}
> testAuditAllowedStat[1](org.apache.hadoop.hdfs.server.namenode.TestAuditLogs)
> Time elapsed: 6.677 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:92)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertNotNull(Assert.java:526)
> at org.junit.Assert.assertNotNull(Assert.java:537)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.verifyAuditLogsRepeat(TestAuditLogs.java:312)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.verifyAuditLogs(TestAuditLogs.java:295)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.testAuditAllowedStat(TestAuditLogs.java:163)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Resolved] (HDFS-5829) TestDNFencingWithReplication fails on branch2
[
https://issues.apache.org/jira/browse/HDFS-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai resolved HDFS-5829.
-
Resolution: Cannot Reproduce
> TestDNFencingWithReplication fails on branch2
> -
>
> Key: HDFS-5829
> URL: https://issues.apache.org/jira/browse/HDFS-5829
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Mit Desai
>
> {noformat}
> Running org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.097 sec <<<
> FAILURE!
> testFencingStress(org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication)
> Time elapsed: 6 sec <<< ERROR!
> java.lang.ExceptionInInitializerError
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication.(TestDNFencingWithReplication.java:49)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881305#comment-13881305
]
Arpit Agarwal commented on HDFS-5776:
-
{quote}
I don't see why we wouldn't expose this setting. It doesn't give the client the
ability to do anything bad it couldn't already do. You can already try to open
a zillion files at once in order to attack the NameNode / DataNodes. Preventing
denial-of-service attacks is not currently something we try to do. And in the
future, if we ever do try to prevent denial-of-service attacks, I don't think
having hedged reads makes that any more or less difficult than it would
otherwise be.
{quote}
[~cmccabe] I am thinking of carelessly configured settings, not a deliberate
dos.
> Support 'hedged' reads in DFSClient
> ---
>
> Key: HDFS-5776
> URL: https://issues.apache.org/jira/browse/HDFS-5776
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt,
> HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776-v8.txt,
> HDFS-5776.txt
>
>
> This is a placeholder of hdfs related stuff backport from
> https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" &
> "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read
> ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we
> could export the interested metric valus into client system(e.g. HBase's
> regionserver metric).
> The core logic is in pread code path, we decide to goto the original
> fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per
> the above config items.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HDFS-5830) WebHdfsFileSystem.getFileBlockLocations throws IllegalArgumentException when accessing another cluster.
Yongjun Zhang created HDFS-5830: --- Summary: WebHdfsFileSystem.getFileBlockLocations throws IllegalArgumentException when accessing another cluster. Key: HDFS-5830 URL: https://issues.apache.org/jira/browse/HDFS-5830 Project: Hadoop HDFS Issue Type: Bug Components: caching, hdfs-client Affects Versions: 2.3.0 Reporter: Yongjun Zhang Assignee: Yongjun Zhang WebHdfsFileSystem.getFileBlockLocations throws IllegalArgumentException when accessing a another cluster (that doesn't have caching support). java.lang.IllegalArgumentException: cachedLocs should not be null, use a different constructor at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) at org.apache.hadoop.hdfs.protocol.LocatedBlock.(LocatedBlock.java:79) at org.apache.hadoop.hdfs.web.JsonUtil.toLocatedBlock(JsonUtil.java:414) at org.apache.hadoop.hdfs.web.JsonUtil.toLocatedBlockList(JsonUtil.java:446) at org.apache.hadoop.hdfs.web.JsonUtil.toLocatedBlocks(JsonUtil.java:479) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileBlockLocations(WebHdfsFileSystem.java:1067) at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1812) at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1797) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5771) Track progress when loading fsimage
[ https://issues.apache.org/jira/browse/HDFS-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5771: - Attachment: HDFS-5771.001.patch Rebased > Track progress when loading fsimage > --- > > Key: HDFS-5771 > URL: https://issues.apache.org/jira/browse/HDFS-5771 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-5698 (FSImage in protobuf) >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5771.000.patch, HDFS-5771.001.patch > > > The old code that loads the fsimage tracks the progress during loading. This > jira proposes to implement the same functionality in the new code which > serializes the fsimage using protobuf.. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881281#comment-13881281 ] stack commented on HDFS-5776: - So, to enable, we set DFS_DFSCLIENT_HEDGED_READ_THREADPOOL_SIZE in DN config. Should the number of threads in the hbase case be greater than NUMBER_OF_HBASE_OPEN_FILES (though this is most often an unknown number, one that is changing over the life over the hbase process, and up in the thousands frequently)? Otherwise we could set it some 'sensible' number like 16 and then just watch the metrics this patch also adds. If we are too often running the requests in the current thread because the executor has none to spare then we can up the number of pool threads (though it requires a DN restart, a PITA)? That should work for the first cut at this feature. nit: You could declare and assign in the one go rather than postpone the assign to the constructor: HEDGED_READ_METRIC = new DFSHedgedReadMetrics(); What is your thinking regards the boolean enabling/disabling hedge reads in DFSClient [~xieliang007]? On the one hand, there is a problem where the setting of pool size is done in DN config yet we have enable/disable hedge reads in the API; if the DN config has a pool size set to 0 then hedged reads are off (as was noted above), and though we may 'enable' hedge reads in the API, we won't be getting the behaviour we think we should be getting. On the other hand, it looks like this boolean could be used 'conserving' resources disabling hedged reads on a per request basis though hedged reads have been marked globally 'on' in the DN? Is that your thinking? I'm inclined to agree with the previous reviewers that this may verge on the 'exotic'. For the first cut at this feature, lets have a global on/off switch with number of threads being the means of constraining how much hedged reading we do? Otherwise patch looks great to me. > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776-v8.txt, > HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5829) TestDNFencingWithReplication fails on branch2
Mit Desai created HDFS-5829:
---
Summary: TestDNFencingWithReplication fails on branch2
Key: HDFS-5829
URL: https://issues.apache.org/jira/browse/HDFS-5829
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Mit Desai
{noformat}
Running org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.097 sec <<<
FAILURE!
testFencingStress(org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication)
Time elapsed: 6 sec <<< ERROR!
java.lang.ExceptionInInitializerError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
at
org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication.(TestDNFencingWithReplication.java:49)
... 28 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Assigned] (HDFS-5827) Children are not inheriting parent's default ACLs
[ https://issues.apache.org/jira/browse/HDFS-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reassigned HDFS-5827: --- Assignee: Chris Nauroth Assigning to myself for verification as part of working on HDFS-5616. > Children are not inheriting parent's default ACLs > - > > Key: HDFS-5827 > URL: https://issues.apache.org/jira/browse/HDFS-5827 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Vinay >Assignee: Chris Nauroth > > Children are not inheriting the parent's default ACLs on creation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5702) FsShell Cli: Add XML based End-to-End test for getfacl and setfacl commands
[
https://issues.apache.org/jira/browse/HDFS-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881220#comment-13881220
]
Chris Nauroth commented on HDFS-5702:
-
Excellent work. Thanks for adding these tests! As mentioned on HDFS-5827, the
2 tests failures are expected, because default ACL handling is not yet
implemented in the NameNode (tracked in HDFS-5616). I'm planning on committing
the failing tests anyway, which is acceptable as a temporary state while we're
isolated on a feature branch.
Just a few minor comments:
# It looks like {{TestAclCLI#fs}} isn't needed for anything, so let's remove
that and the import of {{FileSystem}}.
# Let's add 3 more tests:
## getfacl -R
## setfacl -R
## setfacl --set (replacing the whole ACL)
After that, I think this will be ready to commit.
> FsShell Cli: Add XML based End-to-End test for getfacl and setfacl commands
> ---
>
> Key: HDFS-5702
> URL: https://issues.apache.org/jira/browse/HDFS-5702
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode, security
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5702.patch
>
>
> FsShell Cli: Add XML based End-to-End test for getfacl and setfacl commands
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5827) Children are not inheriting parent's default ACLs
[ https://issues.apache.org/jira/browse/HDFS-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881194#comment-13881194 ] Chris Nauroth commented on HDFS-5827: - Thanks, Vinay. You're right that this is incorrect behavior. This is happening because default ACL handling hasn't been implemented on the NameNode side yet. This is tracked in HDFS-5616, and I'm planning on getting it done in the next few days. Until then, this particular test won't pass. Do you mind if we resolve this as duplicate? For HDFS-5702, I think we can just commit the failing tests, and then I'll verify later that my HDFS-5616 patch fixes them. We're isolated on a feature branch, so having a couple of failing tests for a few days won't harm anyone else. Sound good? > Children are not inheriting parent's default ACLs > - > > Key: HDFS-5827 > URL: https://issues.apache.org/jira/browse/HDFS-5827 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Vinay > > Children are not inheriting the parent's default ACLs on creation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (HDFS-5616) NameNode: implement default ACL handling.
[ https://issues.apache.org/jira/browse/HDFS-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reassigned HDFS-5616: --- Assignee: Chris Nauroth > NameNode: implement default ACL handling. > - > > Key: HDFS-5616 > URL: https://issues.apache.org/jira/browse/HDFS-5616 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > > Implement and test handling of default ACLs within NameNode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5828) BlockPlacementPolicyWithNodeGroup can place multiple replicas on the same node group when dfs.namenode.avoid.write.stale.datanode is true
Buddy created HDFS-5828: --- Summary: BlockPlacementPolicyWithNodeGroup can place multiple replicas on the same node group when dfs.namenode.avoid.write.stale.datanode is true Key: HDFS-5828 URL: https://issues.apache.org/jira/browse/HDFS-5828 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0 Reporter: Buddy When placing replicas using the replica placement policy BlockPlacementPolicyWithNodeGroup, the number of targets returned should be less than or equal to the number of node groups and no node group should get two replicas of the same block. The Junit test TestReplicationPolicyWithNodeGroup.testChooseMoreTargetsThanNodeGroups verifies this. However, if the conf property "dfs.namenode.avoid.write.stale.datanode" is set to true, then block placement policy will return more targets than node groups when the number of replicas requested exceeds the number of node groups. This can be seen by putting: CONF.setBoolean(DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY, true); in the setup method for TestReplicationPolicyWithNodeGroup. This will cause testChooseMoreTargetsThanNodeGroups to fail. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880975#comment-13880975 ] Uma Maheswara Rao G commented on HDFS-5343: --- Seems like jenkins url not accessible. Any one knows, whether jenkins down? > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > Attachments: HDFS-5343-0003.patch, HDFS-5343-0004.patch, > HDFS-5343-0005.patch, HDFS-5343-0006.patch, HDFS-5343-002.patch > > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5702) FsShell Cli: Add XML based End-to-End test for getfacl and setfacl commands
[ https://issues.apache.org/jira/browse/HDFS-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5702: Attachment: HDFS-5702.patch Attaching the patch with some of the initial tests. Please review Among added tests last 2 tests will fail due to HDFS-5827. Please suggest if some more tests needs to be added. > FsShell Cli: Add XML based End-to-End test for getfacl and setfacl commands > --- > > Key: HDFS-5702 > URL: https://issues.apache.org/jira/browse/HDFS-5702 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode, security >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5702.patch > > > FsShell Cli: Add XML based End-to-End test for getfacl and setfacl commands -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5827) Children are not inheriting parent's default ACLs
[ https://issues.apache.org/jira/browse/HDFS-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880942#comment-13880942 ] Vinay commented on HDFS-5827: - Hi [~cnauroth], Could you take a look at this. This I found while writing xml tests. Please correct me if I am wrong. > Children are not inheriting parent's default ACLs > - > > Key: HDFS-5827 > URL: https://issues.apache.org/jira/browse/HDFS-5827 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Vinay > > Children are not inheriting the parent's default ACLs on creation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5827) Children are not inheriting parent's default ACLs
[
https://issues.apache.org/jira/browse/HDFS-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinay updated HDFS-5827:
Description:
Children are not inheriting the parent's default ACLs on creation.
was:
Children are not inheriting the parent's default ACLs on creation.
Following is the ACLs on parent and child in HDFS
{noformat}# file: /dir1
# owner: vinay
# group: supergroup
user::rwx
mask::r-x
other::r-x
default:user::rwx
default:user:charlie:r-x
default:group::r-x
default:group:admin:rwx
default:mask::rwx
default:other::r-x
# file: /dir1/dir2
# owner: vinay
# group: supergroup
user::rwx
group::r-x
other::r-x
# file: /dir1/file
# owner: vinay
# group: supergroup
user::rw-
group::r--
other::r--{noformat}
Following is the output in linux ACL
# file: testAcl
# owner: vinay
# group: users
user::rwx
user:vinay:r--
group::r-x
group:users:r-x
mask::r-x
other::r-x
default:user::rwx
default:user:vinay:r-x
default:group::r-x
default:group:users:rwx
default:mask::rwx
default:other::r-x
# file: testAcl/hello
# owner: vinay
# group: users
user::rw-
user:vinay:r-x #effective:r--
group::r-x #effective:r--
group:users:rwx #effective:rw-
mask::rw-
other::r--
{noformat}
> Children are not inheriting parent's default ACLs
> -
>
> Key: HDFS-5827
> URL: https://issues.apache.org/jira/browse/HDFS-5827
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Vinay
>
> Children are not inheriting the parent's default ACLs on creation.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5827) Children are not inheriting parent's default ACLs
[
https://issues.apache.org/jira/browse/HDFS-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880940#comment-13880940
]
Vinay commented on HDFS-5827:
-
Following is the ACLs on parent and child in HDFS
{noformat}# file: /dir1
# owner: vinay
# group: supergroup
user::rwx
mask::r-x
other::r-x
default:user::rwx
default:user:charlie:r-x
default:group::r-x
default:group:admin:rwx
default:mask::rwx
default:other::r-x
# file: /dir1/dir2
# owner: vinay
# group: supergroup
user::rwx
group::r-x
other::r-x
# file: /dir1/file
# owner: vinay
# group: supergroup
user::rw-
group::r--
other::r--{noformat}
Following is the output in linux ACL
{noformat}
# file: testAcl
# owner: vinay
# group: users
user::rwx
user:vinay:r--
group::r-x
group:users:r-x
mask::r-x
other::r-x
default:user::rwx
default:user:vinay:r-x
default:group::r-x
default:group:users:rwx
default:mask::rwx
default:other::r-x
# file: testAcl/hello
# owner: vinay
# group: users
user::rw-
user:vinay:r-x #effective:r--
group::r-x #effective:r--
group:users:rwx #effective:rw-
mask::rw-
other::r--
{noformat}
> Children are not inheriting parent's default ACLs
> -
>
> Key: HDFS-5827
> URL: https://issues.apache.org/jira/browse/HDFS-5827
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Vinay
>
> Children are not inheriting the parent's default ACLs on creation.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HDFS-5827) Children are not inheriting parent's default ACLs
Vinay created HDFS-5827:
---
Summary: Children are not inheriting parent's default ACLs
Key: HDFS-5827
URL: https://issues.apache.org/jira/browse/HDFS-5827
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Vinay
Children are not inheriting the parent's default ACLs on creation.
Following is the ACLs on parent and child in HDFS
{noformat}# file: /dir1
# owner: vinay
# group: supergroup
user::rwx
mask::r-x
other::r-x
default:user::rwx
default:user:charlie:r-x
default:group::r-x
default:group:admin:rwx
default:mask::rwx
default:other::r-x
# file: /dir1/dir2
# owner: vinay
# group: supergroup
user::rwx
group::r-x
other::r-x
# file: /dir1/file
# owner: vinay
# group: supergroup
user::rw-
group::r--
other::r--{noformat}
Following is the output in linux ACL
# file: testAcl
# owner: vinay
# group: users
user::rwx
user:vinay:r--
group::r-x
group:users:r-x
mask::r-x
other::r-x
default:user::rwx
default:user:vinay:r-x
default:group::r-x
default:group:users:rwx
default:mask::rwx
default:other::r-x
# file: testAcl/hello
# owner: vinay
# group: users
user::rw-
user:vinay:r-x #effective:r--
group::r-x #effective:r--
group:users:rwx #effective:rw-
mask::rw-
other::r--
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880893#comment-13880893 ] Uma Maheswara Rao G commented on HDFS-5343: --- +1, Patch looks good. Pending jenkins report. > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > Attachments: HDFS-5343-0003.patch, HDFS-5343-0004.patch, > HDFS-5343-0005.patch, HDFS-5343-0006.patch, HDFS-5343-002.patch > > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-5343: -- Status: Patch Available (was: Open) > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > Attachments: HDFS-5343-0003.patch, HDFS-5343-0004.patch, > HDFS-5343-0005.patch, HDFS-5343-0006.patch, HDFS-5343-002.patch > > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sathish updated HDFS-5343: -- Attachment: HDFS-5343-0006.patch I updated the patch as per your comments,please review the patch > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > Attachments: HDFS-5343-0003.patch, HDFS-5343-0004.patch, > HDFS-5343-0005.patch, HDFS-5343-0006.patch, HDFS-5343-002.patch > > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[
https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880862#comment-13880862
]
Uma Maheswara Rao G commented on HDFS-5343:
---
{code}
ToolRunner.run(conf, shell, new String[] { "-cat",
+"/TestSnapshotFileLength/sub1/.snapshot/snapshot1/file1" });
+assertEquals(bao.size(), BLOCKSIZE);
+System.setOut(psBackup);
{code}
here System.setOut must be in try finally. Keep ToolRunner execution in try and
keep System.setOut in finally. Previous comment said about unnecessary catching
of exception.
> When cat command is issued on snapshot files getting unexpected result
> --
>
> Key: HDFS-5343
> URL: https://issues.apache.org/jira/browse/HDFS-5343
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: sathish
>Assignee: sathish
> Attachments: HDFS-5343-0003.patch, HDFS-5343-0004.patch,
> HDFS-5343-0005.patch, HDFS-5343-002.patch
>
>
> first if we create one file with some file length and take the snapshot of
> that file,and again append some data through append method to that file,then
> if we do cat command operation on snapshot of that file,in general it should
> dispaly the data what we added with create operation,but it is displaying the
> total data i.e. create +_ appended data.
> but if we do the same operation and if we read the contents of snapshot file
> through input stream it is just displaying the data created in snapshoted
> files.
> in this the behaviour of cat command and reading through inputstream is
> getting different
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sathish updated HDFS-5343: -- Attachment: HDFS-5343-0005.patch please review the patch > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > Attachments: HDFS-5343-0003.patch, HDFS-5343-0004.patch, > HDFS-5343-0005.patch, HDFS-5343-002.patch > > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880859#comment-13880859 ] sathish commented on HDFS-5343: --- yes correct ,no need to catch the exception here,in the previous patch itself i updated that comment but unfortunately it came in latest patch. I will update my patch. > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > Attachments: HDFS-5343-0003.patch, HDFS-5343-0004.patch, > HDFS-5343-002.patch > > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[
https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880849#comment-13880849
]
Uma Maheswara Rao G commented on HDFS-5343:
---
Thanks for the update.
Taking from my previous comment:
{code}
try {
+ToolRunner.run(conf, shell, new String[] { "-cat",
+"/TestSnapshotFileLength/sub1/.snapshot/snapshot1/file1" });
+} catch (Exception e) {
+e.printStackTrace();
+}
{code}
I think you need not catch exception here right?
Could you please answer above question?
> When cat command is issued on snapshot files getting unexpected result
> --
>
> Key: HDFS-5343
> URL: https://issues.apache.org/jira/browse/HDFS-5343
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: sathish
>Assignee: sathish
> Attachments: HDFS-5343-0003.patch, HDFS-5343-0004.patch,
> HDFS-5343-002.patch
>
>
> first if we create one file with some file length and take the snapshot of
> that file,and again append some data through append method to that file,then
> if we do cat command operation on snapshot of that file,in general it should
> dispaly the data what we added with create operation,but it is displaying the
> total data i.e. create +_ appended data.
> but if we do the same operation and if we read the contents of snapshot file
> through input stream it is just displaying the data created in snapshoted
> files.
> in this the behaviour of cat command and reading through inputstream is
> getting different
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880830#comment-13880830 ] sathish commented on HDFS-5343: --- Thanks UMA for reviewing the patch. I updated the patch as per your comments.please review the patch > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > Attachments: HDFS-5343-0003.patch, HDFS-5343-0004.patch, > HDFS-5343-002.patch > > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sathish updated HDFS-5343: -- Attachment: HDFS-5343-0004.patch > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > Attachments: HDFS-5343-0003.patch, HDFS-5343-0004.patch, > HDFS-5343-002.patch > > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5588) Incorrect values of trash configuration and trash emptier state in namenode log
[
https://issues.apache.org/jira/browse/HDFS-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HDFS-5588:
--
Resolution: Duplicate
Release Note: This has already been fixed by HDFS-5560
Status: Resolved (was: Patch Available)
> Incorrect values of trash configuration and trash emptier state in namenode
> log
> ---
>
> Key: HDFS-5588
> URL: https://issues.apache.org/jira/browse/HDFS-5588
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.2.0
>Reporter: Adam Kawa
> Attachments: HDFS-5588-1.patch
>
>
> Values of trash configuration and trash emptier state in namenode log are
> displayed in milliseconds (but it should be seconds). This is very confusing
> because it makes you feel that Trash does not remove data or the meaning of
> configuration settings changed.
> {code}
> // org.apache.hadoop.fs.TrashPolicyDefault
> @Override
> public void initialize(Configuration conf, FileSystem fs, Path home) {
> ...
> this.deletionInterval = (long)(conf.getFloat(
> FS_TRASH_INTERVAL_KEY, FS_TRASH_INTERVAL_DEFAULT)
> * MSECS_PER_MINUTE);
> this.emptierInterval = (long)(conf.getFloat(
> FS_TRASH_CHECKPOINT_INTERVAL_KEY,
> FS_TRASH_CHECKPOINT_INTERVAL_DEFAULT)
> * MSECS_PER_MINUTE);
> LOG.info("Namenode trash configuration: Deletion interval = " +
> this.deletionInterval + " minutes, Emptier interval = " +
> this.emptierInterval + " minutes.");
>}
> {code}
> this.deletionInterval and this.emptierInterval are in miliseconds, but the
> LOG.info tells that they are in minutes.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
