[jira] [Updated] (HDFS-4066) Fix compile warnings for libwebhdfs
[ https://issues.apache.org/jira/browse/HDFS-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4066: Fix Version/s: (was: 2.6.0) 2.7.0 > Fix compile warnings for libwebhdfs > --- > > Key: HDFS-4066 > URL: https://issues.apache.org/jira/browse/HDFS-4066 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.7.0 > > > The compile of libwebhdfs still generates a bunch of warnings, which need to > be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7067) ClassCastException while using a key created by keytool to create encryption zone.
[ https://issues.apache.org/jira/browse/HDFS-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-7067: Fix Version/s: (was: 2.6.0) 2.7.0 > ClassCastException while using a key created by keytool to create encryption > zone. > --- > > Key: HDFS-7067 > URL: https://issues.apache.org/jira/browse/HDFS-7067 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Yi Yao >Priority: Minor > Fix For: 2.7.0 > > > I'm using transparent encryption. If I create a key for KMS keystore via > keytool and use the key to create an encryption zone. I get a > ClassCastException rather than an exception with decent error message. I know > we should use 'hadoop key create' to create a key. It's better to provide an > decent error message to remind user to use the right way to create a KMS key. > [LOG] > ERROR[user=hdfs] Method:'GET' Exception:'java.lang.ClassCastException: > javax.crypto.spec.SecretKeySpec cannot be cast to > org.apache.hadoop.crypto.key.JavaKeyStoreProvider$KeyMetadata' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-5687) Problem in accessing NN JSP page
[ https://issues.apache.org/jira/browse/HDFS-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5687: Fix Version/s: (was: 2.6.0) 2.7.0 > Problem in accessing NN JSP page > > > Key: HDFS-5687 > URL: https://issues.apache.org/jira/browse/HDFS-5687 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: sathish >Assignee: sathish >Priority: Minor > Fix For: 2.7.0 > > Attachments: HDFS-5687-0001.patch > > > In NN UI page After clicking the browse File System page,from that page,if > you click GO Back TO DFS HOME ICon it is not accessing the dfshealth.jsp page > NN http URL is http://nnaddr///nninfoaddr/dfshealth.jsp,it is coming like > this,due to this i think it is not browsing that page > It should be http://nninfoaddr/dfshealth.jsp/ like this -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6694) TestPipelinesFailover.testPipelineRecoveryStress tests fail intermittently with various symptoms
[ https://issues.apache.org/jira/browse/HDFS-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-6694: Fix Version/s: (was: 2.6.0) 2.7.0 > TestPipelinesFailover.testPipelineRecoveryStress tests fail intermittently > with various symptoms > > > Key: HDFS-6694 > URL: https://issues.apache.org/jira/browse/HDFS-6694 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Critical > Fix For: 2.7.0 > > Attachments: HDFS-6694.001.dbg.patch, HDFS-6694.001.dbg.patch, > HDFS-6694.001.dbg.patch, HDFS-6694.002.dbg.patch, > org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover-output.txt, > org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover-output.txt, > org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.txt > > > TestPipelinesFailover.testPipelineRecoveryStress tests fail intermittently > with various symptoms. Typical failures are described in first comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6940) Initial refactoring to allow ConsensusNode implementation
[ https://issues.apache.org/jira/browse/HDFS-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-6940: Fix Version/s: (was: 2.6.0) 2.7.0 > Initial refactoring to allow ConsensusNode implementation > - > > Key: HDFS-6940 > URL: https://issues.apache.org/jira/browse/HDFS-6940 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.6-alpha, 2.5.0 >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko > Fix For: 2.7.0 > > Attachments: HDFS-6940.patch > > > Minor refactoring of FSNamesystem to open private methods that are needed for > CNode implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7273) Add start and stop wrapper scripts for mover
[ https://issues.apache.org/jira/browse/HDFS-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-7273: Fix Version/s: (was: 2.6.0) 2.7.0 > Add start and stop wrapper scripts for mover > > > Key: HDFS-7273 > URL: https://issues.apache.org/jira/browse/HDFS-7273 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Fix For: 2.7.0 > > > Similar to balancer , we need start/stop mover scripts to run data migration > tool as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6300) Allows to run multiple balancer simultaneously
[
https://issues.apache.org/jira/browse/HDFS-6300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-6300:
Fix Version/s: (was: 2.6.0)
2.7.0
> Allows to run multiple balancer simultaneously
> --
>
> Key: HDFS-6300
> URL: https://issues.apache.org/jira/browse/HDFS-6300
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: 2.7.0
>
> Attachments: HDFS-6300.patch
>
>
> Javadoc of "Balancer.java" says, it will not allow to run second balancer if
> the first one is in progress. But I've noticed multiple can run together and
> balancer.id implementation is not safe guarding.
> {code}
> * Another balancer is running. Exiting...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3907) Allow multiple users for local block readers
[
https://issues.apache.org/jira/browse/HDFS-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3907:
Fix Version/s: (was: 2.6.0)
2.7.0
> Allow multiple users for local block readers
>
>
> Key: HDFS-3907
> URL: https://issues.apache.org/jira/browse/HDFS-3907
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.0.0, 2.0.0-alpha
>Reporter: Eli Collins
>Assignee: Eli Collins
> Fix For: 2.7.0
>
> Attachments: hdfs-3907.txt
>
>
> The {{dfs.block.local-path-access.user}} config added in HDFS-2246 only
> supports a single user, however as long as blocks are group readable by more
> than one user the feature could be used by multiple users, to support this we
> just need to allow both to be configured. In practice this allows us to also
> support HBase where the client (RS) runs as the hbase system user and the DN
> runs as hdfs system user. I think this should work secure as well since we're
> not using impersonation in the HBase case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-5160) MiniDFSCluster webui does not work
[
https://issues.apache.org/jira/browse/HDFS-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-5160:
Fix Version/s: (was: 2.6.0)
2.7.0
> MiniDFSCluster webui does not work
> --
>
> Key: HDFS-5160
> URL: https://issues.apache.org/jira/browse/HDFS-5160
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.1.0-beta
>Reporter: Alejandro Abdelnur
> Fix For: 2.7.0
>
>
> The webui does not work, when going to http://localhost:50070 you get:
> {code}
> Directory: /
> webapps/ 102 bytes Sep 4, 2013 9:32:55 AM
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3761) Uses of AuthenticatedURL should set the configured timeouts
[
https://issues.apache.org/jira/browse/HDFS-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3761:
Fix Version/s: (was: 2.6.0)
2.7.0
> Uses of AuthenticatedURL should set the configured timeouts
> ---
>
> Key: HDFS-3761
> URL: https://issues.apache.org/jira/browse/HDFS-3761
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 2.0.2-alpha
>Reporter: Alejandro Abdelnur
>Priority: Critical
> Fix For: 2.7.0
>
>
> Similar to {{URLUtils}}, the {{AuthenticatedURL}} should set the configured
> timeouts in URL connection instances. HADOOP-8644 is introducing a
> {{ConnectionConfigurator}} interface to the {{AuthenticatedURL}} class, it
> should be done via that interface.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6833) DirectoryScanner should not register a deleting block with memory of DataNode
[
https://issues.apache.org/jira/browse/HDFS-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-6833:
Target Version/s: 2.6.1
> DirectoryScanner should not register a deleting block with memory of DataNode
> -
>
> Key: HDFS-6833
> URL: https://issues.apache.org/jira/browse/HDFS-6833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, 2.5.0, 2.5.1
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Critical
> Attachments: HDFS-6833-6-2.patch, HDFS-6833-6-3.patch,
> HDFS-6833-6.patch, HDFS-6833-7-2.patch, HDFS-6833-7.patch, HDFS-6833.8.patch,
> HDFS-6833.9.patch, HDFS-6833.patch, HDFS-6833.patch, HDFS-6833.patch,
> HDFS-6833.patch, HDFS-6833.patch
>
>
> When a block is deleted in DataNode, the following messages are usually
> output.
> {code}
> 2014-08-07 17:53:11,606 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Scheduling blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> for deletion
> 2014-08-07 17:53:11,617 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> {code}
> However, DirectoryScanner may be executed when DataNode deletes the block in
> the current implementation. And the following messsages are output.
> {code}
> 2014-08-07 17:53:30,519 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Scheduling blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> for deletion
> 2014-08-07 17:53:31,426 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1887080305-172.28.0.101-1407398838872 Total blocks: 1, missing metadata
> files:0, missing block files:0, missing blocks in memory:1, mismatched
> blocks:0
> 2014-08-07 17:53:31,426 WARN
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added
> missing block to memory FinalizedReplica, blk_1073741825_1001, FINALIZED
> getNumBytes() = 21230663
> getBytesOnDisk() = 21230663
> getVisibleLength()= 21230663
> getVolume() = /hadoop/data1/dfs/data/current
> getBlockFile()=
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> unlinked =false
> 2014-08-07 17:53:31,531 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> {code}
> Deleting block information is registered in DataNode's memory.
> And when DataNode sends a block report, NameNode receives wrong block
> information.
> For example, when we execute recommission or change the number of
> replication, NameNode may delete the right block as "ExcessReplicate" by this
> problem.
> And "Under-Replicated Blocks" and "Missing Blocks" occur.
> When DataNode run DirectoryScanner, DataNode should not register a deleting
> block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7274) Disable SSLv3 in HttpFS
[
https://issues.apache.org/jira/browse/HDFS-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-7274:
Fix Version/s: 2.6.0
> Disable SSLv3 in HttpFS
> ---
>
> Key: HDFS-7274
> URL: https://issues.apache.org/jira/browse/HDFS-7274
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Fix For: 2.6.0, 2.5.2
>
> Attachments: HDFS-7274.patch, HDFS-7274.patch
>
>
> We should disable SSLv3 in HttpFS to protect against the POODLEbleed
> vulnerability.
> See
> [CVE-2014-3566|http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-3566]
> We have {{sslProtocol="TLS"}} set to only allow TLS in ssl-server.xml, but
> when I checked, I could still connect with SSLv3. There documentation is
> somewhat unclear in the tomcat configs between {{sslProtocol}},
> {{sslProtocols}}, and {{sslEnabledProtocols}} and what each value they take
> does exactly. From what I can gather, {{sslProtocol="TLS"}} actually
> includes SSLv3 and the only way to fix this is to explicitly list which TLS
> versions we support.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7391) Renable SSLv2Hello in HttpFS
[ https://issues.apache.org/jira/browse/HDFS-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-7391: Resolution: Fixed Fix Version/s: 2.6.0 Status: Resolved (was: Patch Available) I just committed this to branch-2.6. Thanks [~rkanter]! [~kasha] - I wasn't sure if you wanted this in branch-2.5, so I haven't cherry-picked it. Please do so if you want to. Thanks. > Renable SSLv2Hello in HttpFS > > > Key: HDFS-7391 > URL: https://issues.apache.org/jira/browse/HDFS-7391 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.6.0, 2.5.2 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Fix For: 2.6.0 > > Attachments: HDFS-7391-branch-2.5.patch, HDFS-7391.patch > > > We should re-enable "SSLv2Hello", which is required for older clients (e.g. > Java 6 with openssl 0.9.8x) so they can't connect without it. Just to be > clear, it does not mean SSLv2, which is insecure. > I couldn't simply do an addendum patch on HDFS-7274 because it's already been > closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7391) Renable SSLv2Hello in HttpFS
[ https://issues.apache.org/jira/browse/HDFS-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209148#comment-14209148 ] Arun C Murthy commented on HDFS-7391: - [~kasha] you just saved me some typing chores, I'll take care of this. Thanks! > Renable SSLv2Hello in HttpFS > > > Key: HDFS-7391 > URL: https://issues.apache.org/jira/browse/HDFS-7391 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.6.0, 2.5.2 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: HDFS-7391-branch-2.5.patch, HDFS-7391.patch > > > We should re-enable "SSLv2Hello", which is required for older clients (e.g. > Java 6 with openssl 0.9.8x) so they can't connect without it. Just to be > clear, it does not mean SSLv2, which is insecure. > I couldn't simply do an addendum patch on HDFS-7274 because it's already been > closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7391) Renable SSLv2Hello in HttpFS
[ https://issues.apache.org/jira/browse/HDFS-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209030#comment-14209030 ] Arun C Murthy commented on HDFS-7391: - Sounds good, thanks [~rkanter] and [~ywskycn]! I'll commit both shortly for RC1. > Renable SSLv2Hello in HttpFS > > > Key: HDFS-7391 > URL: https://issues.apache.org/jira/browse/HDFS-7391 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.6.0, 2.5.2 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: HDFS-7391-branch-2.5.patch, HDFS-7391.patch > > > We should re-enable "SSLv2Hello", which is required for older clients (e.g. > Java 6 with openssl 0.9.8x) so they can't connect without it. Just to be > clear, it does not mean SSLv2, which is insecure. > I couldn't simply do an addendum patch on HDFS-7274 because it's already been > closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7391) Renable SSLv2Hello in HttpFS
[ https://issues.apache.org/jira/browse/HDFS-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208818#comment-14208818 ] Arun C Murthy commented on HDFS-7391: - [~rkanter] - Is this and the addendum patch for HADOOP-11217 the only ones for SSLv2Hello? I thought there were 3? I'll commit both later today if there are no objections; others please commit to branch-2/branch-2.6/branch-2.6.0 if you get to it before me. Thanks. > Renable SSLv2Hello in HttpFS > > > Key: HDFS-7391 > URL: https://issues.apache.org/jira/browse/HDFS-7391 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.6.0, 2.5.2 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: HDFS-7391-branch-2.5.patch, HDFS-7391.patch > > > We should re-enable "SSLv2Hello", which is required for older clients (e.g. > Java 6 with openssl 0.9.8x) so they can't connect without it. Just to be > clear, it does not mean SSLv2, which is insecure. > I couldn't simply do an addendum patch on HDFS-7274 because it's already been > closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7383) DataNode.requestShortCircuitFdsForRead may throw NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204245#comment-14204245
]
Arun C Murthy commented on HDFS-7383:
-
[~sureshms] this might be bad enough that we want it in 2.6? Thanks.
> DataNode.requestShortCircuitFdsForRead may throw NullPointerException
> -
>
> Key: HDFS-7383
> URL: https://issues.apache.org/jira/browse/HDFS-7383
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: 2.7.0
>
> Attachments: h7383_20141108.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.requestShortCircuitFdsForRead(DataNode.java:1525)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitFds(DataXceiver.java:286)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitFds(Receiver.java:185)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:89)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6573) TestDeadDatanode#testDeadDatanode fails if debug flag is turned on for the unit test
[ https://issues.apache.org/jira/browse/HDFS-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-6573: Fix Version/s: (was: 2.4.1) 2.5.0 > TestDeadDatanode#testDeadDatanode fails if debug flag is turned on for the > unit test > > > Key: HDFS-6573 > URL: https://issues.apache.org/jira/browse/HDFS-6573 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.2.0, 2.3.0, 2.4.0 >Reporter: Jinghui Wang >Assignee: Jinghui Wang >Priority: Minor > Fix For: 2.5.0 > > Attachments: HDFS-6573.patch > > > o.a.h.hdfs.server.namenode.NameNodeRpcServer#blockReport calls a function in > for debug mode, which uses the StorageBlockReport data passed in by the test > TestDeadDatanode#testDeadDatanode. The report block list is an long array of > size 3 in the test, but the blockReport expects the format of long array of > size 5. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6527) Edit log corruption due to defered INode removal
[
https://issues.apache.org/jira/browse/HDFS-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039716#comment-14039716
]
Arun C Murthy commented on HDFS-6527:
-
Changed fix version to 2.4.1 since it got merged in for rc1.
> Edit log corruption due to defered INode removal
>
>
> Key: HDFS-6527
> URL: https://issues.apache.org/jira/browse/HDFS-6527
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Fix For: 2.4.1
>
> Attachments: HDFS-6527.branch-2.4.patch, HDFS-6527.trunk.patch,
> HDFS-6527.v2.patch, HDFS-6527.v3.patch, HDFS-6527.v4.patch, HDFS-6527.v5.patch
>
>
> We have seen a SBN crashing with the following error:
> {panel}
> \[Edit log tailer\] ERROR namenode.FSEditLogLoader:
> Encountered exception on operation AddBlockOp
> [path=/xxx,
> penultimateBlock=NULL, lastBlock=blk_111_111, RpcClientId=,
> RpcCallId=-2]
> java.io.FileNotFoundException: File does not exist: /xxx
> {panel}
> This was caused by the deferred removal of deleted inodes from the inode map.
> Since getAdditionalBlock() acquires FSN read lock and then write lock, a
> deletion can happen in between. Because of deferred inode removal outside FSN
> write lock, getAdditionalBlock() can get the deleted inode from the inode map
> with FSN write lock held. This allow addition of a block to a deleted file.
> As a result, the edit log will contain OP_ADD, OP_DELETE, followed by
> OP_ADD_BLOCK. This cannot be replayed by NN, so NN doesn't start up or SBN
> crashes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6527) Edit log corruption due to defered INode removal
[
https://issues.apache.org/jira/browse/HDFS-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-6527:
Fix Version/s: (was: 2.5.0)
2.4.1
> Edit log corruption due to defered INode removal
>
>
> Key: HDFS-6527
> URL: https://issues.apache.org/jira/browse/HDFS-6527
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Fix For: 2.4.1
>
> Attachments: HDFS-6527.branch-2.4.patch, HDFS-6527.trunk.patch,
> HDFS-6527.v2.patch, HDFS-6527.v3.patch, HDFS-6527.v4.patch, HDFS-6527.v5.patch
>
>
> We have seen a SBN crashing with the following error:
> {panel}
> \[Edit log tailer\] ERROR namenode.FSEditLogLoader:
> Encountered exception on operation AddBlockOp
> [path=/xxx,
> penultimateBlock=NULL, lastBlock=blk_111_111, RpcClientId=,
> RpcCallId=-2]
> java.io.FileNotFoundException: File does not exist: /xxx
> {panel}
> This was caused by the deferred removal of deleted inodes from the inode map.
> Since getAdditionalBlock() acquires FSN read lock and then write lock, a
> deletion can happen in between. Because of deferred inode removal outside FSN
> write lock, getAdditionalBlock() can get the deleted inode from the inode map
> with FSN write lock held. This allow addition of a block to a deleted file.
> As a result, the edit log will contain OP_ADD, OP_DELETE, followed by
> OP_ADD_BLOCK. This cannot be replayed by NN, so NN doesn't start up or SBN
> crashes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-3761) Uses of AuthenticatedURL should set the configured timeouts
[
https://issues.apache.org/jira/browse/HDFS-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3761:
Fix Version/s: (was: 2.4.0)
2.5.0
> Uses of AuthenticatedURL should set the configured timeouts
> ---
>
> Key: HDFS-3761
> URL: https://issues.apache.org/jira/browse/HDFS-3761
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 2.0.2-alpha
>Reporter: Alejandro Abdelnur
>Priority: Critical
> Fix For: 2.5.0
>
>
> Similar to {{URLUtils}}, the {{AuthenticatedURL}} should set the configured
> timeouts in URL connection instances. HADOOP-8644 is introducing a
> {{ConnectionConfigurator}} interface to the {{AuthenticatedURL}} class, it
> should be done via that interface.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-3828) Block Scanner rescans blocks too frequently
[
https://issues.apache.org/jira/browse/HDFS-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3828:
Fix Version/s: (was: 2.4.0)
2.5.0
> Block Scanner rescans blocks too frequently
> ---
>
> Key: HDFS-3828
> URL: https://issues.apache.org/jira/browse/HDFS-3828
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Fix For: 2.5.0
>
> Attachments: hdfs-3828-1.txt, hdfs-3828-2.txt, hdfs-3828-3.txt,
> hdfs3828.txt
>
>
> {{BlockPoolSliceScanner#scan}} calls cleanUp every time it's invoked from
> {{DataBlockScanner#run}} via {{scanBlockPoolSlice}}. But cleanUp
> unconditionally roll()s the verificationLogs, so after two iterations we have
> lost the first iteration of block verification times. As a result a cluster
> with just one block repeatedly rescans it every 10 seconds:
> {noformat}
> 2012-08-16 15:59:57,884 INFO datanode.BlockPoolSliceScanner
> (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for
> BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> 2012-08-16 16:00:07,904 INFO datanode.BlockPoolSliceScanner
> (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for
> BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> 2012-08-16 16:00:17,925 INFO datanode.BlockPoolSliceScanner
> (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for
> BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> {noformat}
> {quote}
> To fix this, we need to avoid roll()ing the logs multiple times per period.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-3592) libhdfs should expose ClientProtocol::mkdirs2
[ https://issues.apache.org/jira/browse/HDFS-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-3592: Fix Version/s: (was: 2.4.0) 2.5.0 > libhdfs should expose ClientProtocol::mkdirs2 > - > > Key: HDFS-3592 > URL: https://issues.apache.org/jira/browse/HDFS-3592 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 3.0.0, 2.5.0 > > > It would be nice if libhdfs exposed mkdirs2. This version of mkdirs is much > more verbose about any errors that occur-- it throws AccessControlException, > FileAlreadyExists, FileNotFoundException, ParentNotDirectoryException, etc. > The original mkdirs just throws IOException if anything goes wrong. > For something like fuse_dfs, it is very important to return the correct errno > code when an error has occurred. mkdirs2 would allow us to do that. > I'm not sure if we should just change hdfsMkdirs to use mkdirs2, or add an > hdfsMkdirs2. Probably the latter, but the former course would maintain "bug > compatibility" with ancient releases-- if that is important. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-3907) Allow multiple users for local block readers
[
https://issues.apache.org/jira/browse/HDFS-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3907:
Fix Version/s: (was: 2.4.0)
2.5.0
> Allow multiple users for local block readers
>
>
> Key: HDFS-3907
> URL: https://issues.apache.org/jira/browse/HDFS-3907
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.0.0, 2.0.0-alpha
>Reporter: Eli Collins
>Assignee: Eli Collins
> Fix For: 2.5.0
>
> Attachments: hdfs-3907.txt
>
>
> The {{dfs.block.local-path-access.user}} config added in HDFS-2246 only
> supports a single user, however as long as blocks are group readable by more
> than one user the feature could be used by multiple users, to support this we
> just need to allow both to be configured. In practice this allows us to also
> support HBase where the client (RS) runs as the hbase system user and the DN
> runs as hdfs system user. I think this should work secure as well since we're
> not using impersonation in the HBase case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-3988) HttpFS can't do GETDELEGATIONTOKEN without a prior authenticated request
[ https://issues.apache.org/jira/browse/HDFS-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-3988: Fix Version/s: (was: 2.4.0) 2.5.0 > HttpFS can't do GETDELEGATIONTOKEN without a prior authenticated request > > > Key: HDFS-3988 > URL: https://issues.apache.org/jira/browse/HDFS-3988 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.5.0 > > > A request to obtain a delegation token cannot it initiate an authentication > sequence, it must be accompanied by an auth cookie obtained in a prev request > using a different operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-4754) Add an API in the namenode to mark a datanode as stale
[ https://issues.apache.org/jira/browse/HDFS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4754: Fix Version/s: (was: 2.4.0) 2.5.0 > Add an API in the namenode to mark a datanode as stale > -- > > Key: HDFS-4754 > URL: https://issues.apache.org/jira/browse/HDFS-4754 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Critical > Fix For: 3.0.0, 2.5.0 > > Attachments: 4754.v1.patch, 4754.v2.patch, 4754.v4.patch, > 4754.v4.patch > > > There is a detection of the stale datanodes in HDFS since HDFS-3703, with a > timeout, defaulted to 30s. > There are two reasons to add an API to mark a node as stale even if the > timeout is not yet reached: > 1) ZooKeeper can detect that a client is dead at any moment. So, for HBase, > we sometimes start the recovery before a node is marked staled. (even with > reasonable settings as: stale: 20s; HBase ZK timeout: 30s > 2) Some third parties could detect that a node is dead before the timeout, > hence saving us the cost of retrying. An example or such hw is Arista, > presented here by [~tsuna] > http://tsunanet.net/~tsuna/fsf-hbase-meetup-april13.pdf, and confirmed in > HBASE-6290. > As usual, even if the node is dead it can comeback before the 10 minutes > limit. So I would propose to set a timebound. The API would be > namenode.markStale(String ipAddress, int port, long durationInMs); > After durationInMs, the namenode would again rely only on its heartbeat to > decide. > Thoughts? > If there is no objections, and if nobody in the hdfs dev team has the time to > spend some time on it, I will give it a try for branch 2 & 3. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs
[ https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4489: Fix Version/s: (was: 2.4.0) 2.5.0 > Use InodeID as as an identifier of a file in HDFS protocols and APIs > > > Key: HDFS-4489 > URL: https://issues.apache.org/jira/browse/HDFS-4489 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.5.0 > > Attachments: 4434.optimized.patch > > > The benefit of using InodeID to uniquely identify a file can be multiple > folds. Here are a few of them: > 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, > HDFS-4437. > 2. modification checks in tools like distcp. Since a file could have been > replaced or renamed to, the file name and size combination is no t reliable, > but the combination of file id and size is unique. > 3. id based protocol support (e.g., NFS) > 4. to make the pluggable block placement policy use fileid instead of > filename (HDFS-385). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5160) MiniDFSCluster webui does not work
[
https://issues.apache.org/jira/browse/HDFS-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-5160:
Fix Version/s: (was: 2.4.0)
2.5.0
> MiniDFSCluster webui does not work
> --
>
> Key: HDFS-5160
> URL: https://issues.apache.org/jira/browse/HDFS-5160
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.1.0-beta
>Reporter: Alejandro Abdelnur
> Fix For: 2.5.0
>
>
> The webui does not work, when going to http://localhost:50070 you get:
> {code}
> Directory: /
> webapps/ 102 bytes Sep 4, 2013 9:32:55 AM
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-4066) Fix compile warnings for libwebhdfs
[ https://issues.apache.org/jira/browse/HDFS-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4066: Fix Version/s: (was: 2.4.0) 2.5.0 > Fix compile warnings for libwebhdfs > --- > > Key: HDFS-4066 > URL: https://issues.apache.org/jira/browse/HDFS-4066 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.5.0 > > > The compile of libwebhdfs still generates a bunch of warnings, which need to > be fixed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-4198) the shell script error for Cygwin on windows7
[ https://issues.apache.org/jira/browse/HDFS-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4198: Fix Version/s: (was: 2.4.0) 2.5.0 > the shell script error for Cygwin on windows7 > - > > Key: HDFS-4198 > URL: https://issues.apache.org/jira/browse/HDFS-4198 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.2-alpha > Environment: windows7 ,cygwin. >Reporter: Han Hui Wen > Fix For: 2.5.0 > > > See the following > [comment|https://issues.apache.org/jira/browse/HDFS-4198?focusedCommentId=13498818&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13498818] > for detailed description. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5098) Enhance FileSystem.Statistics to have locality information
[ https://issues.apache.org/jira/browse/HDFS-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5098: Fix Version/s: (was: 2.4.0) 2.5.0 > Enhance FileSystem.Statistics to have locality information > -- > > Key: HDFS-5098 > URL: https://issues.apache.org/jira/browse/HDFS-5098 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Suresh Srinivas > Fix For: 2.5.0 > > > Currently in MR/Tez we dont have a good and accurate means to detect how much > the the IO was actually done locally. Getting this information from the > source of truth would be much better. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-4286) Changes from BOOKKEEPER-203 broken capability of including bookkeeper-server jar in hidden package of BKJM
[ https://issues.apache.org/jira/browse/HDFS-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4286: Fix Version/s: (was: 2.4.0) 2.5.0 > Changes from BOOKKEEPER-203 broken capability of including bookkeeper-server > jar in hidden package of BKJM > -- > > Key: HDFS-4286 > URL: https://issues.apache.org/jira/browse/HDFS-4286 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Vinayakumar B > Fix For: 3.0.0, 2.5.0 > > > BOOKKEEPER-203 introduced changes to LedgerLayout to include > ManagerFactoryClass instead of ManagerFactoryName. > So because of this, BKJM cannot shade the bookkeeper-server jar inside BKJM > jar > LAYOUT znode created by BookieServer is not readable by the BKJM as it have > classes in hidden packages. (same problem vice versa) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5687) Problem in accessing NN JSP page
[ https://issues.apache.org/jira/browse/HDFS-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5687: Fix Version/s: (was: 2.4.0) 2.5.0 > Problem in accessing NN JSP page > > > Key: HDFS-5687 > URL: https://issues.apache.org/jira/browse/HDFS-5687 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: sathish >Assignee: sathish >Priority: Minor > Fix For: 2.5.0 > > Attachments: HDFS-5687-0001.patch > > > In NN UI page After clicking the browse File System page,from that page,if > you click GO Back TO DFS HOME ICon it is not accessing the dfshealth.jsp page > NN http URL is http://nnaddr///nninfoaddr/dfshealth.jsp,it is coming like > this,due to this i think it is not browsing that page > It should be http://nninfoaddr/dfshealth.jsp/ like this -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961906#comment-13961906 ] Arun C Murthy commented on HDFS-6143: - One concern: what is the impact on compatibility with 2.2 etc? > WebHdfsFileSystem open should throw FileNotFoundException for non-existing > paths > > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Attachments: HDFS-6143-branch-2.4.0.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v01.patch, HDFS-6143.v01.patch, > HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch, > HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch > > > WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles > non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6143) (WebHdfs|Hftp)FileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955014#comment-13955014 ] Arun C Murthy commented on HDFS-6143: - [~jira.shegalov] Saw this too late - I'm moving this to 2.4.1, let's try get this in asap. I have rc0 going out, we can re-target 2.4.0 if I need to spin another rc. Thanks. > (WebHdfs|Hftp)FileSystem open should throw FileNotFoundException for > non-existing paths > --- > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Attachments: HDFS-6143.v01.patch, HDFS-6143.v02.patch, > HDFS-6143.v03.patch, HDFS-6143.v04.patch > > > HftpFileSystem.open incorrectly handles non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6143) (WebHdfs|Hftp)FileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-6143: Target Version/s: 2.4.1 (was: 2.4.0) > (WebHdfs|Hftp)FileSystem open should throw FileNotFoundException for > non-existing paths > --- > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Attachments: HDFS-6143.v01.patch, HDFS-6143.v02.patch, > HDFS-6143.v03.patch, HDFS-6143.v04.patch > > > HftpFileSystem.open incorrectly handles non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4564: Resolution: Fixed Fix Version/s: 2.4.0 Status: Resolved (was: Patch Available) I just committed this. Thanks [~daryn] (and [~jingzhao] for the review)! > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Fix For: 2.4.0 > > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.branch-23.patch, HDFS-4564.patch, HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954997#comment-13954997 ] Arun C Murthy commented on HDFS-4564: - I'm going to commit this, based on [~jingzhao]'s +1 - I'd like to roll 2.4 rc0 asap since my availability next few days is limited with the Hadoop Summit. > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.branch-23.patch, HDFS-4564.patch, HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954374#comment-13954374 ] Arun C Murthy commented on HDFS-4564: - [~daryn] - Can you pls clarify some of the above review comments? I'd like to move fwd with 2.4 ASAP. Tx! > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.branch-23.patch, HDFS-4564.patch, HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5840) Follow-up to HDFS-5138 to improve error handling during partial upgrade failures
[ https://issues.apache.org/jira/browse/HDFS-5840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944061#comment-13944061 ] Arun C Murthy commented on HDFS-5840: - [~sureshms] / [~atm] - Should this be a blocker for 2.4? Thanks. > Follow-up to HDFS-5138 to improve error handling during partial upgrade > failures > > > Key: HDFS-5840 > URL: https://issues.apache.org/jira/browse/HDFS-5840 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 3.0.0 > > Attachments: HDFS-5840.patch > > > Suresh posted some good comment in HDFS-5138 after that patch had already > been committed to trunk. This JIRA is to address those. See the first comment > of this JIRA for the full content of the review. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5138) Support HDFS upgrade in HA
[ https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944060#comment-13944060 ] Arun C Murthy commented on HDFS-5138: - [~jingzhao] / [~sureshms] - Should we close this one and mark HDFS-6135 as the blocker? Thanks. > Support HDFS upgrade in HA > -- > > Key: HDFS-5138 > URL: https://issues.apache.org/jira/browse/HDFS-5138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Kihwal Lee >Assignee: Aaron T. Myers >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > hdfs-5138-branch-2.txt > > > With HA enabled, NN wo't start with "-upgrade". Since there has been a layout > version change between 2.0.x and 2.1.x, starting NN in upgrade mode was > necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way > to get around this was to disable HA and upgrade. > The NN and the cluster cannot be flipped back to HA until the upgrade is > finalized. If HA is disabled only on NN for layout upgrade and HA is turned > back on without involving DNs, things will work, but finaliizeUpgrade won't > work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade > snapshots won't get removed. > We will need a different ways of doing layout upgrade and upgrade snapshot. > I am marking this as a 2.1.1-beta blocker based on feedback from others. If > there is a reasonable workaround that does not increase maintenance window > greatly, we can lower its priority from blocker to critical. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932008#comment-13932008 ] Arun C Murthy commented on HDFS-4564: - How is this looking [~daryn]? Thanks. (Doing a pass over 2.4 blockers) > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.branch-23.patch, HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-3907) Allow multiple users for local block readers
[
https://issues.apache.org/jira/browse/HDFS-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3907:
Fix Version/s: (was: 2.3.0)
2.4.0
> Allow multiple users for local block readers
>
>
> Key: HDFS-3907
> URL: https://issues.apache.org/jira/browse/HDFS-3907
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.0.0, 2.0.0-alpha
>Reporter: Eli Collins
>Assignee: Eli Collins
> Fix For: 2.4.0
>
> Attachments: hdfs-3907.txt
>
>
> The {{dfs.block.local-path-access.user}} config added in HDFS-2246 only
> supports a single user, however as long as blocks are group readable by more
> than one user the feature could be used by multiple users, to support this we
> just need to allow both to be configured. In practice this allows us to also
> support HBase where the client (RS) runs as the hbase system user and the DN
> runs as hdfs system user. I think this should work secure as well since we're
> not using impersonation in the HBase case.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-4198) the shell script error for Cygwin on windows7
[ https://issues.apache.org/jira/browse/HDFS-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4198: Fix Version/s: (was: 2.3.0) 2.4.0 > the shell script error for Cygwin on windows7 > - > > Key: HDFS-4198 > URL: https://issues.apache.org/jira/browse/HDFS-4198 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.2-alpha > Environment: windows7 ,cygwin. >Reporter: Han Hui Wen > Fix For: 2.4.0 > > > See the following > [comment|https://issues.apache.org/jira/browse/HDFS-4198?focusedCommentId=13498818&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13498818] > for detailed description. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5522) Datanode disk error check may be incorrectly skipped
[
https://issues.apache.org/jira/browse/HDFS-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-5522:
Fix Version/s: (was: 2.3.0)
2.4.0
> Datanode disk error check may be incorrectly skipped
>
>
> Key: HDFS-5522
> URL: https://issues.apache.org/jira/browse/HDFS-5522
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.9, 2.2.0
>Reporter: Kihwal Lee
> Fix For: 0.23.11, 2.4.0
>
>
> After HDFS-4581 and HDFS-4699, {{checkDiskError()}} is not called when
> network errors occur during processing data node requests. This appears to
> create problems when a disk is having problems, but not failing I/O soon.
> If I/O hangs for a long time, network read/write may timeout first and the
> peer may close the connection. Although the error was caused by a faulty
> local disk, disk check is not being carried out in this case.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-4066) Fix compile warnings for libwebhdfs
[ https://issues.apache.org/jira/browse/HDFS-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4066: Fix Version/s: (was: 2.3.0) 2.4.0 > Fix compile warnings for libwebhdfs > --- > > Key: HDFS-4066 > URL: https://issues.apache.org/jira/browse/HDFS-4066 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.4.0 > > > The compile of libwebhdfs still generates a bunch of warnings, which need to > be fixed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5160) MiniDFSCluster webui does not work
[
https://issues.apache.org/jira/browse/HDFS-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-5160:
Fix Version/s: (was: 2.3.0)
2.4.0
> MiniDFSCluster webui does not work
> --
>
> Key: HDFS-5160
> URL: https://issues.apache.org/jira/browse/HDFS-5160
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.1.0-beta
>Reporter: Alejandro Abdelnur
> Fix For: 2.4.0
>
>
> The webui does not work, when going to http://localhost:50070 you get:
> {code}
> Directory: /
> webapps/ 102 bytes Sep 4, 2013 9:32:55 AM
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5244) TestNNStorageRetentionManager#testPurgeMultipleDirs fails because incorrectly expects Hashmap values to have order.
[ https://issues.apache.org/jira/browse/HDFS-5244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5244: Fix Version/s: (was: 2.3.0) 2.4.0 > TestNNStorageRetentionManager#testPurgeMultipleDirs fails because incorrectly > expects Hashmap values to have order. > > > Key: HDFS-5244 > URL: https://issues.apache.org/jira/browse/HDFS-5244 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM java 1.6 >Reporter: Jinghui Wang >Assignee: Jinghui Wang > Fix For: 3.0.0, 2.1.0-beta, 2.4.0 > > Attachments: HDFS-5244.patch > > > The test o.a.h.hdfs.server.namenode.TestNNStorageRetentionManager uses a > HashMap(dirRoots) to store the root storages to be mocked for the purging > test, which does not have any predictable order. The directories needs be > purged are stored in a LinkedHashSet, which has a predictable order. So, when > the directories get mocked for the test, they could be already out of > the order that they were added. Thus, the order that the directories were > actually purged and the order of them being added to the LinkedHashList could > be different and cause the test to fail. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-3761) Uses of AuthenticatedURL should set the configured timeouts
[
https://issues.apache.org/jira/browse/HDFS-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3761:
Fix Version/s: (was: 2.3.0)
2.4.0
> Uses of AuthenticatedURL should set the configured timeouts
> ---
>
> Key: HDFS-3761
> URL: https://issues.apache.org/jira/browse/HDFS-3761
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 2.0.2-alpha
>Reporter: Alejandro Abdelnur
>Priority: Critical
> Fix For: 2.4.0
>
>
> Similar to {{URLUtils}}, the {{AuthenticatedURL}} should set the configured
> timeouts in URL connection instances. HADOOP-8644 is introducing a
> {{ConnectionConfigurator}} interface to the {{AuthenticatedURL}} class, it
> should be done via that interface.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-3592) libhdfs should expose ClientProtocol::mkdirs2
[ https://issues.apache.org/jira/browse/HDFS-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-3592: Fix Version/s: (was: 2.3.0) 2.4.0 > libhdfs should expose ClientProtocol::mkdirs2 > - > > Key: HDFS-3592 > URL: https://issues.apache.org/jira/browse/HDFS-3592 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 3.0.0, 2.4.0 > > > It would be nice if libhdfs exposed mkdirs2. This version of mkdirs is much > more verbose about any errors that occur-- it throws AccessControlException, > FileAlreadyExists, FileNotFoundException, ParentNotDirectoryException, etc. > The original mkdirs just throws IOException if anything goes wrong. > For something like fuse_dfs, it is very important to return the correct errno > code when an error has occurred. mkdirs2 would allow us to do that. > I'm not sure if we should just change hdfsMkdirs to use mkdirs2, or add an > hdfsMkdirs2. Probably the latter, but the former course would maintain "bug > compatibility" with ancient releases-- if that is important. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs
[ https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4489: Fix Version/s: (was: 2.3.0) 2.4.0 > Use InodeID as as an identifier of a file in HDFS protocols and APIs > > > Key: HDFS-4489 > URL: https://issues.apache.org/jira/browse/HDFS-4489 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.4.0 > > Attachments: 4434.optimized.patch > > > The benefit of using InodeID to uniquely identify a file can be multiple > folds. Here are a few of them: > 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, > HDFS-4437. > 2. modification checks in tools like distcp. Since a file could have been > replaced or renamed to, the file name and size combination is no t reliable, > but the combination of file id and size is unique. > 3. id based protocol support (e.g., NFS) > 4. to make the pluggable block placement policy use fileid instead of > filename (HDFS-385). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-3988) HttpFS can't do GETDELEGATIONTOKEN without a prior authenticated request
[ https://issues.apache.org/jira/browse/HDFS-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-3988: Fix Version/s: (was: 2.3.0) 2.4.0 > HttpFS can't do GETDELEGATIONTOKEN without a prior authenticated request > > > Key: HDFS-3988 > URL: https://issues.apache.org/jira/browse/HDFS-3988 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.4.0 > > > A request to obtain a delegation token cannot it initiate an authentication > sequence, it must be accompanied by an auth cookie obtained in a prev request > using a different operation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5687) Problem in accessing NN JSP page
[ https://issues.apache.org/jira/browse/HDFS-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5687: Fix Version/s: (was: 2.3.0) 2.4.0 > Problem in accessing NN JSP page > > > Key: HDFS-5687 > URL: https://issues.apache.org/jira/browse/HDFS-5687 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: sathish >Assignee: sathish >Priority: Minor > Fix For: 2.4.0 > > Attachments: HDFS-5687-0001.patch > > > In NN UI page After clicking the browse File System page,from that page,if > you click GO Back TO DFS HOME ICon it is not accessing the dfshealth.jsp page > NN http URL is http://nnaddr///nninfoaddr/dfshealth.jsp,it is coming like > this,due to this i think it is not browsing that page > It should be http://nninfoaddr/dfshealth.jsp/ like this -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-4286) Changes from BOOKKEEPER-203 broken capability of including bookkeeper-server jar in hidden package of BKJM
[ https://issues.apache.org/jira/browse/HDFS-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4286: Fix Version/s: (was: 2.3.0) 2.4.0 > Changes from BOOKKEEPER-203 broken capability of including bookkeeper-server > jar in hidden package of BKJM > -- > > Key: HDFS-4286 > URL: https://issues.apache.org/jira/browse/HDFS-4286 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Vinayakumar B > Fix For: 3.0.0, 2.4.0 > > > BOOKKEEPER-203 introduced changes to LedgerLayout to include > ManagerFactoryClass instead of ManagerFactoryName. > So because of this, BKJM cannot shade the bookkeeper-server jar inside BKJM > jar > LAYOUT znode created by BookieServer is not readable by the BKJM as it have > classes in hidden packages. (same problem vice versa) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-2802) Support for RW/RO snapshots in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-2802: Fix Version/s: (was: 2.3.0) 2.4.0 > Support for RW/RO snapshots in HDFS > --- > > Key: HDFS-2802 > URL: https://issues.apache.org/jira/browse/HDFS-2802 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Hari Mankude >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 2.4.0 > > Attachments: 2802.diff, 2802.patch, 2802.patch, > HDFS-2802-meeting-minutes-121101.txt, HDFS-2802.20121101.patch, > HDFS-2802.branch-2.0510.patch, HDFSSnapshotsDesign.pdf, Snapshots.pdf, > Snapshots20120429.pdf, Snapshots20121018.pdf, Snapshots20121030.pdf, > h2802_20130417.patch, h2802_20130422.patch, h2802_20130423.patch, > h2802_20130425.patch, h2802_20130426.patch, h2802_20130430.patch, > h2802_20130430b.patch, h2802_20130508.patch, h2802_20130509.patch, > snap.patch, snapshot-design.pdf, snapshot-design.tex, snapshot-one-pager.pdf, > snapshot-testplan.pdf > > > Snapshots are point in time images of parts of the filesystem or the entire > filesystem. Snapshots can be a read-only or a read-write point in time copy > of the filesystem. There are several use cases for snapshots in HDFS. I will > post a detailed write-up soon with with more information. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-3746) Invalid counter tag in HDFS balancer which lead to infinite loop
[ https://issues.apache.org/jira/browse/HDFS-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-3746: Fix Version/s: (was: 2.3.0) 2.4.0 > Invalid counter tag in HDFS balancer which lead to infinite loop > > > Key: HDFS-3746 > URL: https://issues.apache.org/jira/browse/HDFS-3746 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: meng gong >Assignee: meng gong > Labels: patch > Fix For: 3.0.0, 2.4.0 > > > Everytime banlancer try to move a block cross the rack. For every > NameNodeConnector it will instance a new banlancer. The tag > notChangedIterations is reseted to 0. Then it won't reach 5 and exit the > thread for not moved in 5 consecutive iterations. This lead to a infinite loop -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-2832: Fix Version/s: (was: 2.3.0) 2.4.0 > Enable support for heterogeneous storages in HDFS > - > > Key: HDFS-2832 > URL: https://issues.apache.org/jira/browse/HDFS-2832 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 3.0.0, 2.4.0 > > Attachments: 20130813-HeterogeneousStorage.pdf, > 20131125-HeterogeneousStorage-TestPlan.pdf, > 20131125-HeterogeneousStorage.pdf, > 20131202-HeterogeneousStorage-TestPlan.pdf, > 20131203-HeterogeneousStorage-TestPlan.pdf, H2832_20131107.patch, > editsStored, editsStored, h2832_20131023.patch, h2832_20131023b.patch, > h2832_20131025.patch, h2832_20131028.patch, h2832_20131028b.patch, > h2832_20131029.patch, h2832_20131103.patch, h2832_20131104.patch, > h2832_20131105.patch, h2832_20131107b.patch, h2832_20131108.patch, > h2832_20131110.patch, h2832_20131110b.patch, h2832_2013.patch, > h2832_20131112.patch, h2832_20131112b.patch, h2832_20131114.patch, > h2832_20131118.patch, h2832_20131119.patch, h2832_20131119b.patch, > h2832_20131121.patch, h2832_20131122.patch, h2832_20131122b.patch, > h2832_20131123.patch, h2832_20131124.patch, h2832_20131202.patch, > h2832_20131203.patch, h2832_20131210.patch, h2832_20131211.patch, > h2832_20131211b.patch, h2832_branch-2_20131226.patch, > h2832_branch-2_20140103.patch > > > HDFS currently supports configuration where storages are a list of > directories. Typically each of these directories correspond to a volume with > its own file system. All these directories are homogeneous and therefore > identified as a single storage at the namenode. I propose, change to the > current model where Datanode * is a * storage, to Datanode * is a collection > * of strorages. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5098) Enhance FileSystem.Statistics to have locality information
[ https://issues.apache.org/jira/browse/HDFS-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5098: Fix Version/s: (was: 2.3.0) 2.4.0 > Enhance FileSystem.Statistics to have locality information > -- > > Key: HDFS-5098 > URL: https://issues.apache.org/jira/browse/HDFS-5098 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Suresh Srinivas > Fix For: 2.4.0 > > > Currently in MR/Tez we dont have a good and accurate means to detect how much > the the IO was actually done locally. Getting this information from the > source of truth would be much better. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-3747) balancer-Test failed because of time out
[ https://issues.apache.org/jira/browse/HDFS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-3747: Fix Version/s: (was: 2.3.0) 2.4.0 > balancer-Test failed because of time out > - > > Key: HDFS-3747 > URL: https://issues.apache.org/jira/browse/HDFS-3747 > Project: Hadoop HDFS > Issue Type: Test > Components: balancer >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: meng gong >Assignee: meng gong > Labels: test > Fix For: 3.0.0, 2.4.0 > > Attachments: TestCaseForBanlancer.java > > > When run a given test case for Banlancer Test. In the test the banlancer > thread try to move some block cross the rack but it can't find any available > blocks in the source rack. Then the thread won't interrupt until the tag > isTimeUp reaches 20min. But maven judges the test failed because the thread > have runned for 15min. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-4754) Add an API in the namenode to mark a datanode as stale
[ https://issues.apache.org/jira/browse/HDFS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4754: Fix Version/s: (was: 2.3.0) 2.4.0 > Add an API in the namenode to mark a datanode as stale > -- > > Key: HDFS-4754 > URL: https://issues.apache.org/jira/browse/HDFS-4754 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Critical > Fix For: 3.0.0, 2.4.0 > > Attachments: 4754.v1.patch, 4754.v2.patch, 4754.v4.patch, > 4754.v4.patch > > > There is a detection of the stale datanodes in HDFS since HDFS-3703, with a > timeout, defaulted to 30s. > There are two reasons to add an API to mark a node as stale even if the > timeout is not yet reached: > 1) ZooKeeper can detect that a client is dead at any moment. So, for HBase, > we sometimes start the recovery before a node is marked staled. (even with > reasonable settings as: stale: 20s; HBase ZK timeout: 30s > 2) Some third parties could detect that a node is dead before the timeout, > hence saving us the cost of retrying. An example or such hw is Arista, > presented here by [~tsuna] > http://tsunanet.net/~tsuna/fsf-hbase-meetup-april13.pdf, and confirmed in > HBASE-6290. > As usual, even if the node is dead it can comeback before the 10 minutes > limit. So I would propose to set a timebound. The API would be > namenode.markStale(String ipAddress, int port, long durationInMs); > After durationInMs, the namenode would again rely only on its heartbeat to > decide. > Thoughts? > If there is no objections, and if nobody in the hdfs dev team has the time to > spend some time on it, I will give it a try for branch 2 & 3. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-3828) Block Scanner rescans blocks too frequently
[
https://issues.apache.org/jira/browse/HDFS-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3828:
Fix Version/s: (was: 2.3.0)
2.4.0
> Block Scanner rescans blocks too frequently
> ---
>
> Key: HDFS-3828
> URL: https://issues.apache.org/jira/browse/HDFS-3828
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Fix For: 2.4.0
>
> Attachments: hdfs-3828-1.txt, hdfs-3828-2.txt, hdfs-3828-3.txt,
> hdfs3828.txt
>
>
> {{BlockPoolSliceScanner#scan}} calls cleanUp every time it's invoked from
> {{DataBlockScanner#run}} via {{scanBlockPoolSlice}}. But cleanUp
> unconditionally roll()s the verificationLogs, so after two iterations we have
> lost the first iteration of block verification times. As a result a cluster
> with just one block repeatedly rescans it every 10 seconds:
> {noformat}
> 2012-08-16 15:59:57,884 INFO datanode.BlockPoolSliceScanner
> (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for
> BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> 2012-08-16 16:00:07,904 INFO datanode.BlockPoolSliceScanner
> (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for
> BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> 2012-08-16 16:00:17,925 INFO datanode.BlockPoolSliceScanner
> (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for
> BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> {noformat}
> {quote}
> To fix this, we need to avoid roll()ing the logs multiple times per period.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13891167#comment-13891167 ] Arun C Murthy commented on HDFS-4564: - Ok, thanks [~daryn]! > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13891086#comment-13891086 ] Arun C Murthy commented on HDFS-4564: - [~daryn] - Is this close? Thanks. > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4564: Target Version/s: 2.3.0 (was: 3.0.0, 2.3.0) > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-4949) Centralized cache management in HDFS
[ https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855882#comment-13855882 ] Arun C Murthy commented on HDFS-4949: - We've been discussing through a proposal where-in we can leverage YARN's resource-management *and* workload-management capabilities (via delegation of resources, in this case RAM to HDFS) to provide a more general cache administration in YARN-1488. > Centralized cache management in HDFS > > > Key: HDFS-4949 > URL: https://issues.apache.org/jira/browse/HDFS-4949 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-4949-consolidated.patch, > caching-design-doc-2013-07-02.pdf, caching-design-doc-2013-08-09.pdf, > caching-design-doc-2013-10-24.pdf, caching-testplan.pdf > > > HDFS currently has no support for managing or exposing in-memory caches at > datanodes. This makes it harder for higher level application frameworks like > Hive, Pig, and Impala to effectively use cluster memory, because they cannot > explicitly cache important datasets or place their tasks for memory locality. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5307) Support both HTTP and HTTPS in jsp pages
[ https://issues.apache.org/jira/browse/HDFS-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5307: Fix Version/s: (was: 2.2.0) 2.2.1 > Support both HTTP and HTTPS in jsp pages > > > Key: HDFS-5307 > URL: https://issues.apache.org/jira/browse/HDFS-5307 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.2.1 > > Attachments: HDFS-5307.000.patch > > > Adopt the new APIs provided by HDFS-5306 to support both HTTP and HTTPS in > the jsp pages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5317: Fix Version/s: (was: 2.2.0) 2.2.1 > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > Fix For: 2.2.1 > > Attachments: HDFS-5317.000.patch, HDFS-5317.001.patch > > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-5291:
Fix Version/s: (was: 2.2.0)
2.2.1
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Fix For: 2.2.1
>
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, HDFS-5291.003.patch, HDFS-5291.003.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new epoch 85
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(249)) - Beginning recovery
> of unclosed segment starting at txid 887112
> 2013-10-02 00:13:49,874 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(258)) - Recovery prepare
> phase complete. Responses:
> IP:8485: segmentState { startTxId: 887112 endTxId: 887531 isInProgress: true
> } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 172.18.145.97:8485: segmentState { startTxId: 887112 endTxId: 887531
> isInProgress: true } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 2013-10-02 00:13:49,875 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recover
> {code}
> And then we get into safemode
> {code}
> Construction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,277 INFO BlockStateChange
> (BlockManager.java:logAddStoredBlock(2237)) - BLOCK* addStoredBlock: blockMap
> updated: IP:1019 is added to blk_IP157{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[IP:1019|RBW],
> ReplicaUnderConstruction[172.18.145.96:1019|RBW], ReplicaUnde
> rConstruction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,279 INFO hdfs.StateChange
> (FSNamesystem.
[jira] [Commented] (HDFS-4817) make HDFS advisory caching configurable on a per-file basis
[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13787917#comment-13787917
]
Arun C Murthy commented on HDFS-4817:
-
[~cmccabe] In future, if you backport patches, please ensure you reflect it
appropriately in all branches.
For e.g.
HDFS-4817. Make HDFS advisory caching configurable on a per-file basis.
(Colin Patrick McCabe)
shows up in Release 2.3.0 as an Improvement in trunk/branch-2 while it shows up
as a New Feature in branch-2.1-beta.
This will reduce the burden on making releases, makes it consistent for folks
to figure release etc. Thanks.
> make HDFS advisory caching configurable on a per-file basis
> ---
>
> Key: HDFS-4817
> URL: https://issues.apache.org/jira/browse/HDFS-4817
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
> HDFS-4817.004.patch, HDFS-4817.006.patch, HDFS-4817.007.patch,
> HDFS-4817.008.patch, HDFS-4817.009.patch, HDFS-4817.010.patch,
> HDFS-4817-b2.1.001.patch
>
>
> HADOOP-7753 and related JIRAs introduced some performance optimizations for
> the DataNode. One of them was readahead. When readahead is enabled, the
> DataNode starts reading the next bytes it thinks it will need in the block
> file, before the client requests them. This helps hide the latency of
> rotational media and send larger reads down to the device. Another
> optimization was "drop-behind." Using this optimization, we could remove
> files from the Linux page cache after they were no longer needed.
> Using {{dfs.datanode.drop.cache.behind.writes}} and
> {{dfs.datanode.drop.cache.behind.reads}} can improve performance
> substantially on many MapReduce jobs. In our internal benchmarks, we have
> seen speedups of 40% on certain workloads. The reason is because if we know
> the block data will not be read again any time soon, keeping it out of memory
> allows more memory to be used by the other processes on the system. See
> HADOOP-7714 for more benchmarks.
> We would like to turn on these configurations on a per-file or per-client
> basis, rather than on the DataNode as a whole. This will allow more users to
> actually make use of them. It would also be good to add unit tests for the
> drop-cache code path, to ensure that it is functioning as we expect.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5251) Race between the initialization of NameNode and the http server
[ https://issues.apache.org/jira/browse/HDFS-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5251: Fix Version/s: (was: 2.2.0) 2.1.2-beta > Race between the initialization of NameNode and the http server > --- > > Key: HDFS-5251 > URL: https://issues.apache.org/jira/browse/HDFS-5251 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.1.2-beta > > Attachments: HDFS-5251.000.patch > > > The constructor of NameNode starts a HTTP server before the FSNameSystem is > initialized. Currently there is a race where the HTTP server can access the > uninitialized namesystem variable, throwing a NullPointerException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5139) Remove redundant -R option from setrep
[ https://issues.apache.org/jira/browse/HDFS-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5139: Fix Version/s: (was: 2.2.0) 2.1.2-beta > Remove redundant -R option from setrep > -- > > Key: HDFS-5139 > URL: https://issues.apache.org/jira/browse/HDFS-5139 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 3.0.0, 1.3.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.1.2-beta > > Attachments: HDFS-5139.01.patch, HDFS-5139.02.patch, > HDFS-5139.03.patch, HDFS-5139.04.patch > > > The -R option to setrep is redundant because it is required for directory > targets and ignored for file targets. > We can just remove the option and make -R the default for directories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4754) Add an API in the namenode to mark a datanode as stale
[ https://issues.apache.org/jira/browse/HDFS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4754: Fix Version/s: (was: 2.1.1-beta) 2.1.2-beta > Add an API in the namenode to mark a datanode as stale > -- > > Key: HDFS-4754 > URL: https://issues.apache.org/jira/browse/HDFS-4754 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Critical > Fix For: 3.0.0, 2.1.2-beta > > Attachments: 4754.v1.patch, 4754.v2.patch, 4754.v4.patch, > 4754.v4.patch > > > There is a detection of the stale datanodes in HDFS since HDFS-3703, with a > timeout, defaulted to 30s. > There are two reasons to add an API to mark a node as stale even if the > timeout is not yet reached: > 1) ZooKeeper can detect that a client is dead at any moment. So, for HBase, > we sometimes start the recovery before a node is marked staled. (even with > reasonable settings as: stale: 20s; HBase ZK timeout: 30s > 2) Some third parties could detect that a node is dead before the timeout, > hence saving us the cost of retrying. An example or such hw is Arista, > presented here by [~tsuna] > http://tsunanet.net/~tsuna/fsf-hbase-meetup-april13.pdf, and confirmed in > HBASE-6290. > As usual, even if the node is dead it can comeback before the 10 minutes > limit. So I would propose to set a timebound. The API would be > namenode.markStale(String ipAddress, int port, long durationInMs); > After durationInMs, the namenode would again rely only on its heartbeat to > decide. > Thoughts? > If there is no objections, and if nobody in the hdfs dev team has the time to > spend some time on it, I will give it a try for branch 2 & 3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5098) Enhance FileSystem.Statistics to have locality information
[ https://issues.apache.org/jira/browse/HDFS-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5098: Fix Version/s: (was: 2.1.1-beta) 2.1.2-beta > Enhance FileSystem.Statistics to have locality information > -- > > Key: HDFS-5098 > URL: https://issues.apache.org/jira/browse/HDFS-5098 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Suresh Srinivas > Fix For: 2.1.2-beta > > > Currently in MR/Tez we dont have a good and accurate means to detect how much > the the IO was actually done locally. Getting this information from the > source of truth would be much better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5244) TestNNStorageRetentionManager#testPurgeMultipleDirs fails because incorrectly expects Hashmap values to have order.
[ https://issues.apache.org/jira/browse/HDFS-5244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5244: Fix Version/s: (was: 2.1.1-beta) 2.1.2-beta > TestNNStorageRetentionManager#testPurgeMultipleDirs fails because incorrectly > expects Hashmap values to have order. > > > Key: HDFS-5244 > URL: https://issues.apache.org/jira/browse/HDFS-5244 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.0-beta, 2.1.2-beta > > Attachments: HDFS-5244.patch > > > The test o.a.h.hdfs.server.namenode.TestNNStorageRetentionManager uses a > HashMap(dirRoots) to store the root storages to be mocked for the purging > test, which does not have any predictable order. The directories needs be > purged are stored in a LinkedHashSet, which has a predictable order. So, when > the directories get mocked for the test, they could be already out of > the order that they were added. Thus, the order that the directories were > actually purged and the order of them being added to the LinkedHashList could > be different and cause the test to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5160) MiniDFSCluster webui does not work
[
https://issues.apache.org/jira/browse/HDFS-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-5160:
Fix Version/s: (was: 2.1.1-beta)
2.1.2-beta
> MiniDFSCluster webui does not work
> --
>
> Key: HDFS-5160
> URL: https://issues.apache.org/jira/browse/HDFS-5160
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.1.0-beta
>Reporter: Alejandro Abdelnur
> Fix For: 2.1.2-beta
>
>
> The webui does not work, when going to http://localhost:50070 you get:
> {code}
> Directory: /
> webapps/ 102 bytes Sep 4, 2013 9:32:55 AM
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5246) Hadoop nfs server binds to port 2049 which is the same as Linux nfs server
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5246: Fix Version/s: (was: 2.1.1-beta) 2.1.2-beta > Hadoop nfs server binds to port 2049 which is the same as Linux nfs server > -- > > Key: HDFS-5246 > URL: https://issues.apache.org/jira/browse/HDFS-5246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.2-beta > > Attachments: HDFS-5246-2.patch, HDFS-5246-3.patch, HDFS-5246.patch > > > Hadoop nfs binds the nfs server to port 2049, > which is also the default port that Linux nfs uses. If Linux nfs is already > running on the machine then Hadoop nfs will not be albe to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5228: Fix Version/s: (was: 2.1.1-beta) 2.1.2-beta > The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw > NPE > > > Key: HDFS-5228 > URL: https://issues.apache.org/jira/browse/HDFS-5228 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.1.0-beta >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Blocker > Fix For: 2.1.2-beta > > Attachments: h5228_20130919.patch, h5228_20130919_test.patch, > HDFS-5228.2.patch > > > Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative > path. Then, it will result a NullPointerException when calling hasNext() > from the RemoteIterator. > This bug was discovered by Arnaud: > http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-5225:
Target Version/s: 2.1.2-beta
> datanode keeps logging the same 'is no longer in the dataset' message over
> and over again
> -
>
> Key: HDFS-5225
> URL: https://issues.apache.org/jira/browse/HDFS-5225
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.1.1-beta
>Reporter: Roman Shaposhnik
>Priority: Blocker
> Attachments: HDFS-5225.1.patch, HDFS-5225-reproduce.1.txt
>
>
> I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
> configuration: 4 nodes fully distributed cluster with security on.
> All of a sudden my DN ate up all of the space in /var/log logging the
> following message repeatedly:
> {noformat}
> 2013-09-18 20:51:12,046 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
> in the dataset
> {noformat}
> It wouldn't answer to a jstack and jstack -F ended up being useless.
> Here's what I was able to find in the NameNode logs regarding this block ID:
> {noformat}
> fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
> 2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
> BP-1884637155-10.224.158.152-1379524544853
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
> 2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.224.158.152:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.34.74.206:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.83.107.80:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,899 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
> newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
> 10.34.74.206:1004, 10.224.158.152:1004],
> clientName=DFSClient_NONMAPREDUCE_-450304083_1)
> 2013-09-18 18:03:17,904 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
> successfully to
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
> 2013-09-18 18:03:18,540 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
> newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
> 10.34.74.206:1004, 10.224.158.152:1004],
> clientName=DFSClient_NONMAPREDUCE_-450304083_1)
> 2013-09-18 18:03:18,548 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
> successfully to
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
> 2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
> blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
> 2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
> blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
> blk_1073742188_1368, blk_1073742189_1371]
> 2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> InvalidateBlocks: ask 10.224.158.152:1004 to delete [blk_1073742177_1353,
> blk_1073742178_1359, blk_1073742179_1355, blk_1073742180_1356,
> blk_1073742181_1358, blk_1073742182_1361, blk_10737
[jira] [Updated] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-5225:
Priority: Blocker (was: Major)
> datanode keeps logging the same 'is no longer in the dataset' message over
> and over again
> -
>
> Key: HDFS-5225
> URL: https://issues.apache.org/jira/browse/HDFS-5225
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.1.1-beta
>Reporter: Roman Shaposhnik
>Priority: Blocker
> Attachments: HDFS-5225.1.patch, HDFS-5225-reproduce.1.txt
>
>
> I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
> configuration: 4 nodes fully distributed cluster with security on.
> All of a sudden my DN ate up all of the space in /var/log logging the
> following message repeatedly:
> {noformat}
> 2013-09-18 20:51:12,046 INFO
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
> in the dataset
> {noformat}
> It wouldn't answer to a jstack and jstack -F ended up being useless.
> Here's what I was able to find in the NameNode logs regarding this block ID:
> {noformat}
> fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
> 2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
> BP-1884637155-10.224.158.152-1379524544853
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
> 2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.224.158.152:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.34.74.206:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.83.107.80:1004 is added to
> blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
> ReplicaUnderConstruction[10.34.74.206:1004|RBW],
> ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
> 2013-09-18 18:03:17,899 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
> newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
> 10.34.74.206:1004, 10.224.158.152:1004],
> clientName=DFSClient_NONMAPREDUCE_-450304083_1)
> 2013-09-18 18:03:17,904 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
> successfully to
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
> 2013-09-18 18:03:18,540 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
> newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
> 10.34.74.206:1004, 10.224.158.152:1004],
> clientName=DFSClient_NONMAPREDUCE_-450304083_1)
> 2013-09-18 18:03:18,548 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
> successfully to
> BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
> 2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
> blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
> 2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
> blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
> blk_1073742188_1368, blk_1073742189_1371]
> 2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> InvalidateBlocks: ask 10.224.158.152:1004 to delete [blk_1073742177_1353,
> blk_1073742178_1359, blk_1073742179_1355, blk_1073742180_1356,
> blk_1073742181_1358, blk_1073742182_1361, blk_10
[jira] [Updated] (HDFS-5138) Support HDFS upgrade in HA
[ https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-5138: Target Version/s: (was: 2.1.1-beta) > Support HDFS upgrade in HA > -- > > Key: HDFS-5138 > URL: https://issues.apache.org/jira/browse/HDFS-5138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Kihwal Lee >Priority: Blocker > > With HA enabled, NN wo't start with "-upgrade". Since there has been a layout > version change between 2.0.x and 2.1.x, starting NN in upgrade mode was > necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way > to get around this was to disable HA and upgrade. > The NN and the cluster cannot be flipped back to HA until the upgrade is > finalized. If HA is disabled only on NN for layout upgrade and HA is turned > back on without involving DNs, things will work, but finaliizeUpgrade won't > work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade > snapshots won't get removed. > We will need a different ways of doing layout upgrade and upgrade snapshot. > I am marking this as a 2.1.1-beta blocker based on feedback from others. If > there is a reasonable workaround that does not increase maintenance window > greatly, we can lower its priority from blocker to critical. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3983) Hftp should support both SPNEGO and KSSL
[
https://issues.apache.org/jira/browse/HDFS-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768351#comment-13768351
]
Arun C Murthy commented on HDFS-3983:
-
Any update on this? Is this ready to go? Thanks.
> Hftp should support both SPNEGO and KSSL
>
>
> Key: HDFS-3983
> URL: https://issues.apache.org/jira/browse/HDFS-3983
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.0.0-alpha
>Reporter: Eli Collins
>Assignee: Eli Collins
>Priority: Blocker
> Attachments: hdfs-3983.txt, hdfs-3983.txt
>
>
> Hftp currently doesn't work against a secure cluster unless you configure
> {{dfs.https.port}} to be the http port, otherwise the client can't fetch
> tokens:
> {noformat}
> $ hadoop fs -ls hftp://c1225.hal.cloudera.com:50070/
> 12/09/26 18:02:00 INFO fs.FileSystem: Couldn't get a delegation token from
> http://c1225.hal.cloudera.com:50470 using http.
> ls: Security enabled but user not authenticated by filter
> {noformat}
> This is due to Hftp still using the https port. Post HDFS-2617 it should use
> the regular http port. Hsftp should still use the secure port, however now
> that we have HADOOP-8581 it's worth considering removing Hsftp entirely. I'll
> start a separate thread about that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4952) dfs -ls hftp:// fails on secure hadoop2 cluster
[ https://issues.apache.org/jira/browse/HDFS-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved HDFS-4952. - Resolution: Fixed Duplicate of HDFS-3983 > dfs -ls hftp:// fails on secure hadoop2 cluster > --- > > Key: HDFS-4952 > URL: https://issues.apache.org/jira/browse/HDFS-4952 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yesha Vora >Priority: Blocker > > Running :hadoop dfs -ls hftp://:/A > WARN fs.FileSystem: Couldn't connect to http://:50470, assuming > security is disabled > ls: Security enabled but user not authenticated by filter -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4952) dfs -ls hftp:// fails on secure hadoop2 cluster
[ https://issues.apache.org/jira/browse/HDFS-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4952: Target Version/s: (was: 2.1.1-beta) > dfs -ls hftp:// fails on secure hadoop2 cluster > --- > > Key: HDFS-4952 > URL: https://issues.apache.org/jira/browse/HDFS-4952 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yesha Vora >Priority: Blocker > > Running :hadoop dfs -ls hftp://:/A > WARN fs.FileSystem: Couldn't connect to http://:50470, assuming > security is disabled > ls: Security enabled but user not authenticated by filter -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5138) Support HDFS upgrade in HA
[ https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765519#comment-13765519 ] Arun C Murthy commented on HDFS-5138: - Guys, any luck? I'd like to get 2.1.1 out soon... thanks! > Support HDFS upgrade in HA > -- > > Key: HDFS-5138 > URL: https://issues.apache.org/jira/browse/HDFS-5138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Kihwal Lee >Priority: Blocker > > With HA enabled, NN wo't start with "-upgrade". Since there has been a layout > version change between 2.0.x and 2.1.x, starting NN in upgrade mode was > necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way > to get around this was to disable HA and upgrade. > The NN and the cluster cannot be flipped back to HA until the upgrade is > finalized. If HA is disabled only on NN for layout upgrade and HA is turned > back on without involving DNs, things will work, but finaliizeUpgrade won't > work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade > snapshots won't get removed. > We will need a different ways of doing layout upgrade and upgrade snapshot. > I am marking this as a 2.1.1-beta blocker based on feedback from others. If > there is a reasonable workaround that does not increase maintenance window > greatly, we can lower its priority from blocker to critical. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3981) access time is set without holding FSNamesystem write lock
[
https://issues.apache.org/jira/browse/HDFS-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3981:
Fix Version/s: (was: 2.1.0-beta)
2.3.0
> access time is set without holding FSNamesystem write lock
> --
>
> Key: HDFS-3981
> URL: https://issues.apache.org/jira/browse/HDFS-3981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 0.23.3, 2.0.3-alpha, 0.23.5
>Reporter: Xiaobo Peng
>Assignee: Xiaobo Peng
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HDFS-3981-branch-0.23.4.patch,
> HDFS-3981-branch-0.23.patch, HDFS-3981-branch-2.patch, HDFS-3981-trunk.patch,
> hdfs-3981.txt
>
>
> Incorrect condition in {{FSNamesystem.getBlockLocatoins()}} can lead to
> updating times without write lock. In most cases this condition will force
> {{FSNamesystem.getBlockLocatoins()}} to hold write lock, even if times do not
> need to be updated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3761) Uses of AuthenticatedURL should set the configured timeouts
[
https://issues.apache.org/jira/browse/HDFS-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3761:
Fix Version/s: (was: 2.1.0-beta)
2.3.0
> Uses of AuthenticatedURL should set the configured timeouts
> ---
>
> Key: HDFS-3761
> URL: https://issues.apache.org/jira/browse/HDFS-3761
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha, 2.0.2-alpha
>Reporter: Alejandro Abdelnur
>Priority: Critical
> Fix For: 2.3.0
>
>
> Similar to {{URLUtils}}, the {{AuthenticatedURL}} should set the configured
> timeouts in URL connection instances. HADOOP-8644 is introducing a
> {{ConnectionConfigurator}} interface to the {{AuthenticatedURL}} class, it
> should be done via that interface.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4198) the shell script error for Cygwin on windows7
[ https://issues.apache.org/jira/browse/HDFS-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4198: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > the shell script error for Cygwin on windows7 > - > > Key: HDFS-4198 > URL: https://issues.apache.org/jira/browse/HDFS-4198 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.2-alpha > Environment: windows7 ,cygwin. >Reporter: Han Hui Wen > Fix For: 2.3.0 > > > See the following > [comment|https://issues.apache.org/jira/browse/HDFS-4198?focusedCommentId=13498818&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13498818] > for detailed description. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4066) Fix compile warnings for libwebhdfs
[ https://issues.apache.org/jira/browse/HDFS-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4066: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > Fix compile warnings for libwebhdfs > --- > > Key: HDFS-4066 > URL: https://issues.apache.org/jira/browse/HDFS-4066 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.3.0 > > > The compile of libwebhdfs still generates a bunch of warnings, which need to > be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4940) namenode OOMs under Bigtop's TestCLI
[ https://issues.apache.org/jira/browse/HDFS-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4940: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > namenode OOMs under Bigtop's TestCLI > > > Key: HDFS-4940 > URL: https://issues.apache.org/jira/browse/HDFS-4940 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik >Priority: Blocker > Fix For: 2.3.0 > > > Bigtop's TestCLI when executed against Hadoop 2.1.0 seems to make it OOM > quite reliably regardless of the heap size settings. I'm attaching a heap > dump URL. Alliteratively anybody can just take Bigtop's tests, compiled them > against Hadoop 2.1.0 bits and try to reproduce it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3691) UserGroupInformation#spawnAutoRenewalThreadForUserCreds tries to renew even if the kerberos ticket cache is non-renewable
[
https://issues.apache.org/jira/browse/HDFS-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3691:
Fix Version/s: (was: 2.1.0-beta)
2.3.0
> UserGroupInformation#spawnAutoRenewalThreadForUserCreds tries to renew even
> if the kerberos ticket cache is non-renewable
> -
>
> Key: HDFS-3691
> URL: https://issues.apache.org/jira/browse/HDFS-3691
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.1-alpha
>Reporter: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.3.0
>
>
> UserGroupInformation#spawnAutoRenewalThreadForUserCreds tries to renew user
> credentials. However, it does this even if the kerberos ticket cache in
> question is non-renewable.
> This leads to an annoying error message being printed out all the time.
> {code}
> cmccabe@keter:/h> klist
> Ticket cache: FILE:/tmp/krb5cc_1014
> Default principal: hdfs/ke...@cloudera.com
> Valid starting ExpiresService principal
> 07/18/12 15:24:15 07/19/12 15:24:13 krbtgt/cloudera@cloudera.com
> {code}
> {code}
> cmccabe@keter:/h> ./bin/hadoop fs -ls /
> 15:21:39,882 WARN UserGroupInformation:739 - Exception encountered while
> running the renewal command. Aborting renew thread.
> org.apache.hadoop.util.Shell$ExitCodeException: kinit: KDC can't fulfill
> requested option while renewing credentials
> Found 3 items
> -rw-r--r-- 3 cmccabe users 0 2012-07-09 17:15 /b
> -rw-r--r-- 3 hdfssupergroup 0 2012-07-09 17:17 /c
> drwxrwxrwx - cmccabe audio 0 2012-07-19 11:25 /tmp
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3828) Block Scanner rescans blocks too frequently
[
https://issues.apache.org/jira/browse/HDFS-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3828:
Fix Version/s: (was: 2.1.0-beta)
2.3.0
> Block Scanner rescans blocks too frequently
> ---
>
> Key: HDFS-3828
> URL: https://issues.apache.org/jira/browse/HDFS-3828
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Fix For: 2.3.0
>
> Attachments: hdfs-3828-1.txt, hdfs-3828-2.txt, hdfs-3828-3.txt,
> hdfs3828.txt
>
>
> {{BlockPoolSliceScanner#scan}} calls cleanUp every time it's invoked from
> {{DataBlockScanner#run}} via {{scanBlockPoolSlice}}. But cleanUp
> unconditionally roll()s the verificationLogs, so after two iterations we have
> lost the first iteration of block verification times. As a result a cluster
> with just one block repeatedly rescans it every 10 seconds:
> {noformat}
> 2012-08-16 15:59:57,884 INFO datanode.BlockPoolSliceScanner
> (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for
> BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> 2012-08-16 16:00:07,904 INFO datanode.BlockPoolSliceScanner
> (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for
> BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> 2012-08-16 16:00:17,925 INFO datanode.BlockPoolSliceScanner
> (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for
> BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> {noformat}
> {quote}
> To fix this, we need to avoid roll()ing the logs multiple times per period.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2802) Support for RW/RO snapshots in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-2802: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > Support for RW/RO snapshots in HDFS > --- > > Key: HDFS-2802 > URL: https://issues.apache.org/jira/browse/HDFS-2802 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Hari Mankude >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 2.3.0 > > Attachments: 2802.diff, 2802.patch, 2802.patch, h2802_20130417.patch, > h2802_20130422.patch, h2802_20130423.patch, h2802_20130425.patch, > h2802_20130426.patch, h2802_20130430b.patch, h2802_20130430.patch, > h2802_20130508.patch, h2802_20130509.patch, HDFS-2802.20121101.patch, > HDFS-2802.branch-2.0510.patch, HDFS-2802-meeting-minutes-121101.txt, > HDFSSnapshotsDesign.pdf, snap.patch, snapshot-design.pdf, > snapshot-design.tex, snapshot-one-pager.pdf, Snapshots20120429.pdf, > Snapshots20121018.pdf, Snapshots20121030.pdf, Snapshots.pdf, > snapshot-testplan.pdf > > > Snapshots are point in time images of parts of the filesystem or the entire > filesystem. Snapshots can be a read-only or a read-write point in time copy > of the filesystem. There are several use cases for snapshots in HDFS. I will > post a detailed write-up soon with with more information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3746) Invalid counter tag in HDFS balancer which lead to infinite loop
[ https://issues.apache.org/jira/browse/HDFS-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-3746: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > Invalid counter tag in HDFS balancer which lead to infinite loop > > > Key: HDFS-3746 > URL: https://issues.apache.org/jira/browse/HDFS-3746 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: meng gong >Assignee: meng gong > Labels: patch > Fix For: 3.0.0, 2.3.0 > > > Everytime banlancer try to move a block cross the rack. For every > NameNodeConnector it will instance a new banlancer. The tag > notChangedIterations is reseted to 0. Then it won't reach 5 and exit the > thread for not moved in 5 consecutive iterations. This lead to a infinite loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3907) Allow multiple users for local block readers
[
https://issues.apache.org/jira/browse/HDFS-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-3907:
Fix Version/s: (was: 2.1.0-beta)
2.3.0
> Allow multiple users for local block readers
>
>
> Key: HDFS-3907
> URL: https://issues.apache.org/jira/browse/HDFS-3907
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.0.0, 2.0.0-alpha
>Reporter: Eli Collins
>Assignee: Eli Collins
> Fix For: 2.3.0
>
> Attachments: hdfs-3907.txt
>
>
> The {{dfs.block.local-path-access.user}} config added in HDFS-2246 only
> supports a single user, however as long as blocks are group readable by more
> than one user the feature could be used by multiple users, to support this we
> just need to allow both to be configured. In practice this allows us to also
> support HBase where the client (RS) runs as the hbase system user and the DN
> runs as hdfs system user. I think this should work secure as well since we're
> not using impersonation in the HBase case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3592) libhdfs should expose ClientProtocol::mkdirs2
[ https://issues.apache.org/jira/browse/HDFS-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-3592: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > libhdfs should expose ClientProtocol::mkdirs2 > - > > Key: HDFS-3592 > URL: https://issues.apache.org/jira/browse/HDFS-3592 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > > It would be nice if libhdfs exposed mkdirs2. This version of mkdirs is much > more verbose about any errors that occur-- it throws AccessControlException, > FileAlreadyExists, FileNotFoundException, ParentNotDirectoryException, etc. > The original mkdirs just throws IOException if anything goes wrong. > For something like fuse_dfs, it is very important to return the correct errno > code when an error has occurred. mkdirs2 would allow us to do that. > I'm not sure if we should just change hdfsMkdirs to use mkdirs2, or add an > hdfsMkdirs2. Probably the latter, but the former course would maintain "bug > compatibility" with ancient releases-- if that is important. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3987) add support for encrypted webhdfs
[ https://issues.apache.org/jira/browse/HDFS-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-3987: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > add support for encrypted webhdfs > - > > Key: HDFS-3987 > URL: https://issues.apache.org/jira/browse/HDFS-3987 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur > Fix For: 2.3.0 > > > This is a follow up of HDFS-3983. > We should have a new filesystem client impl/binding for encrypted WebHDFS, > i.e. *webhdfss://* > On the server side, webhdfs and httpfs we should only need to start the > service on a secured (HTTPS) endpoint. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4286) Changes from BOOKKEEPER-203 broken capability of including bookkeeper-server jar in hidden package of BKJM
[ https://issues.apache.org/jira/browse/HDFS-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4286: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > Changes from BOOKKEEPER-203 broken capability of including bookkeeper-server > jar in hidden package of BKJM > -- > > Key: HDFS-4286 > URL: https://issues.apache.org/jira/browse/HDFS-4286 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Vinay > Fix For: 3.0.0, 2.3.0 > > > BOOKKEEPER-203 introduced changes to LedgerLayout to include > ManagerFactoryClass instead of ManagerFactoryName. > So because of this, BKJM cannot shade the bookkeeper-server jar inside BKJM > jar > LAYOUT znode created by BookieServer is not readable by the BKJM as it have > classes in hidden packages. (same problem vice versa) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3988) HttpFS can't do GETDELEGATIONTOKEN without a prior authenticated request
[ https://issues.apache.org/jira/browse/HDFS-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-3988: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > HttpFS can't do GETDELEGATIONTOKEN without a prior authenticated request > > > Key: HDFS-3988 > URL: https://issues.apache.org/jira/browse/HDFS-3988 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.3.0 > > > A request to obtain a delegation token cannot it initiate an authentication > sequence, it must be accompanied by an auth cookie obtained in a prev request > using a different operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4559) WebHDFS does not allow resolution of Symlinks
[ https://issues.apache.org/jira/browse/HDFS-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4559: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > WebHDFS does not allow resolution of Symlinks > - > > Key: HDFS-4559 > URL: https://issues.apache.org/jira/browse/HDFS-4559 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.3-alpha >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.3.0 > > > WebHDFS allows you to create symlinks via the CREATESYMLINK operation, but > the GETFILEINFO operation specifically calls the getFileInfo() method of the > NameNodeRpcServer which does not resolve symlinks. > I propose adding a parameter to GETFILEINFO such that if true will call > getFileLinkInfo() rather than getFileInfo() which will resolve any symlinks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs
[ https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HDFS-4489: Fix Version/s: (was: 2.1.0-beta) 2.3.0 > Use InodeID as as an identifier of a file in HDFS protocols and APIs > > > Key: HDFS-4489 > URL: https://issues.apache.org/jira/browse/HDFS-4489 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.3.0 > > Attachments: 4434.optimized.patch > > > The benefit of using InodeID to uniquely identify a file can be multiple > folds. Here are a few of them: > 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, > HDFS-4437. > 2. modification checks in tools like distcp. Since a file could have been > replaced or renamed to, the file name and size combination is no t reliable, > but the combination of file id and size is unique. > 3. id based protocol support (e.g., NFS) > 4. to make the pluggable block placement policy use fileid instead of > filename (HDFS-385). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
