[jira] [Updated] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-5431: -- Attachment: hdfs-5431-5.patch Fix tests and findbugs, do some cleanups. support cachepool-based limit management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch, hdfs-5431-4.patch, hdfs-5431-5.patch We should support cachepool-based limit management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5632) Add Snapshot feature to INodeDirectory
[
https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848303#comment-13848303
]
Hadoop QA commented on HDFS-5632:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618695/HDFS-5632.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5720//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5720//console
This message is automatically generated.
Add Snapshot feature to INodeDirectory
--
Key: HDFS-5632
URL: https://issues.apache.org/jira/browse/HDFS-5632
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-5632.000.patch, HDFS-5632.001.patch,
HDFS-5632.002.patch, HDFS-5632.003.patch
We will add snapshot feature to INodeDirectory and remove
INodeDirectoryWithSnapshot in this jira.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5632) Add Snapshot feature to INodeDirectory
[ https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-5632: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) I have committed this. Thanks, Jing! Add Snapshot feature to INodeDirectory -- Key: HDFS-5632 URL: https://issues.apache.org/jira/browse/HDFS-5632 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao Fix For: 3.0.0 Attachments: HDFS-5632.000.patch, HDFS-5632.001.patch, HDFS-5632.002.patch, HDFS-5632.003.patch We will add snapshot feature to INodeDirectory and remove INodeDirectoryWithSnapshot in this jira. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5632) Add Snapshot feature to INodeDirectory
[ https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848309#comment-13848309 ] Hudson commented on HDFS-5632: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4883 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4883/]) HDFS-5632. Flatten INodeDirectory hierarchy: Replace INodeDirectoryWithSnapshot with DirectoryWithSnapshotFeature. Contributed by jing9 (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1550917) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestNestedSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSetQuotaWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotRename.java Add Snapshot feature to INodeDirectory --
[jira] [Commented] (HDFS-5632) Add Snapshot feature to INodeDirectory
[ https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848318#comment-13848318 ] Hudson commented on HDFS-5632: -- FAILURE: Integrated in Hadoop-Yarn-trunk #421 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/421/]) HDFS-5632. Flatten INodeDirectory hierarchy: Replace INodeDirectoryWithSnapshot with DirectoryWithSnapshotFeature. Contributed by jing9 (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1550917) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestNestedSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSetQuotaWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotRename.java Add Snapshot feature to INodeDirectory -- Key:
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848323#comment-13848323
]
Hadoop QA commented on HDFS-5431:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618766/hdfs-5431-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestSafeMode
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5721//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5721//console
This message is automatically generated.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
hdfs-5431-4.patch, hdfs-5431-5.patch
We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5664) try to relieve the BlockReaderLocal read() synchronized hotspot
[
https://issues.apache.org/jira/browse/HDFS-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848340#comment-13848340
]
Liang Xie commented on HDFS-5664:
-
bq. since there would still be a big synchronized on all the
DFSInputStream#read methods which use the BlockReader
This can be fixed by HDFS-1605, e.g. use a read lock for read()
bq. If multiple threads want to read the same file at the same time, they can
open multiple distinct streams for it. At that point, they're not sharing the
same BlockReader, so whether or not BRL is synchronized doesn't matter.
yes, this is a feasible idea.
But in current HBase codebase, we use only one stream(or two streams
considering checksum or not in old version) for one HFile.So seems here is a
critical performance issue. we should try to figure out is it possible to
remove the synchronized keyword in BlockReader or we must consider to use
multiple thread pattern. [~stack], do you familiar with here: why HBase use one
stream always for one HFile in history？
I'll try to understand some background here as well.
try to relieve the BlockReaderLocal read() synchronized hotspot
---
Key: HDFS-5664
URL: https://issues.apache.org/jira/browse/HDFS-5664
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.0.0, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
Current the BlockReaderLocal's read has a synchronized modifier:
{code}
public synchronized int read(byte[] buf, int off, int len) throws IOException
{
{code}
In a HBase physical read heavy cluster, we observed some hotspots from
dfsclient path, the detail strace trace could be found from:
https://issues.apache.org/jira/browse/HDFS-1605?focusedCommentId=13843241page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13843241
I haven't looked into the detail yet, put some raw ideas here firstly:
1) replace synchronized with try lock with timeout pattern, so could
fail-fast, 2) fallback to non-ssr mode if get a local reader lock failed.
There're two suitable scenario at least to remove this hotspot:
1) Local physical read heavy, e.g. HBase block cache miss ratio is high
2) slow/bad disk.
It would be helpful to achive a lower 99th percentile HBase read latency
somehow.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5632) Add Snapshot feature to INodeDirectory
[ https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848354#comment-13848354 ] Hudson commented on HDFS-5632: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1612 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1612/]) HDFS-5632. Flatten INodeDirectory hierarchy: Replace INodeDirectoryWithSnapshot with DirectoryWithSnapshotFeature. Contributed by jing9 (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1550917) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestNestedSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSetQuotaWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotRename.java Add Snapshot feature to INodeDirectory --
[jira] [Commented] (HDFS-5434) Write resiliency for replica count 1
[
https://issues.apache.org/jira/browse/HDFS-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848363#comment-13848363
]
Eric Sirianni commented on HDFS-5434:
-
bq. I sort of assumed when I read this JIRA that Eric was considering
situations where the admin knows that the storage is reliable despite having
only one replica.
Correct - thanks for jumping in.
bq. If the target storage is reliable then what do we gain by adding an extra
replica in the pipeline?
bq. Resiliency against transient network errors.
Yes, network errors, but also host failure. In the architecture we are
targeting, we use:
* RAID for resiliency to disk failure
* Shared storage for resiliency to host failure (this is enabled via an
{{FsDatasetSpi}} plugin that my team is developing)
These combine to make replicaCount=1 a viable alternative for some use cases.
However, the host failure resiliency via shared storage is only applicable once
the block is finalized after the initial write. Therefore, for a being-written
block, this architecture is still susceptible to data loss via host failure.
The solution proposed by this JIRA is to _temporarily_ use replicaCount=2
during the initial write pipeline (for host failure resiliency) and then drop
back to replicaCount=1 post-block-finalize (for storage efficiency).
The initial proposal was to control this in the NameNode by vending a pipeline
of length 2 even if the client requested replicaCount=1. In many ways, this is
a cleaner solution as it more directly models the desired architecture
(replicaCount is always 1, but pipeline length is forced to be 1) . However,
Colin expressed some concerns about overriding the client's request at the
NameNode. We are considering at a client-side only approach as a fallback
alternative. Arpit - do you share Colin's concerns with the server-side
approach?
Write resiliency for replica count 1
Key: HDFS-5434
URL: https://issues.apache.org/jira/browse/HDFS-5434
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.2.0
Reporter: Buddy
Priority: Minor
If a file has a replica count of one, the HDFS client is exposed to write
failures if the data node fails during a write. With a pipeline of size of
one, no recovery is possible if the sole data node dies.
A simple fix is to force a minimum pipeline size of 2, while leaving the
replication count as 1. The implementation for this is fairly non-invasive.
Although the replica count is one, the block will be written to two data
nodes instead of one. If one of the data nodes fails during the write, normal
pipeline recovery will ensure that the write succeeds to the surviving data
node.
The existing code in the name node will prune the extra replica when it
receives the block received reports for the finalized block from both data
nodes. This results in the intended replica count of one for the block.
This behavior should be controlled by a configuration option such as
{{dfs.namenode.minPipelineSize}}.
This behavior can be implemented in {{FSNameSystem.getAdditionalBlock()}} by
ensuring that the pipeline size passed to
{{BlockPlacementPolicy.chooseTarget()}} in the replication parameter is:
{code}
max(replication, ${dfs.namenode.minPipelineSize})
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5632) Add Snapshot feature to INodeDirectory
[ https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848370#comment-13848370 ] Hudson commented on HDFS-5632: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1638 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1638/]) HDFS-5632. Flatten INodeDirectory hierarchy: Replace INodeDirectoryWithSnapshot with DirectoryWithSnapshotFeature. Contributed by jing9 (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1550917) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestNestedSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSetQuotaWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotRename.java Add Snapshot feature to INodeDirectory --
[jira] [Commented] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[ https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848410#comment-13848410 ] Benoy Antony commented on HDFS-5661: AuthFilter.java is used only for webhdfs. While accessing JSP files, AuthenticationFilter is used and AuthenticationFilter does not use delegationToken. Note that the use of IP address while generating redirectURL was introduced with HDFS-5307. It used to be fqdn before. From the HDFS-5307 patch , -String fqdn = canonicalize(chosenNode.getIpAddr()); -String tailUrl = /// + fqdn + : + chosenNode.getInfoPort() + +String tailUrl = /// + JspHelper.Url.authority(req.getScheme(), chosenNode) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address Key: HDFS-5661 URL: https://issues.apache.org/jira/browse/HDFS-5661 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.2.0 Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HDFS-5661.patch If authentication is enabled on the web ui, then a cookie is used to keep track of the authentication information. There is normally a domain associated with the cookie. Since ip address doesn't have any domain , the cookie will not be sent by the browser while making http calls with ip address as the destination server. This will break browsing files system via web ui , if authentication is enabled. Browsing FileSystem via web ui, should use datanode's hostname instead of ip address. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HDFS-5669) Storage#tryLock() should check for null before logging successfull message
Vinay created HDFS-5669:
---
Summary: Storage#tryLock() should check for null before logging
successfull message
Key: HDFS-5669
URL: https://issues.apache.org/jira/browse/HDFS-5669
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Vinay
Assignee: Vinay
In the following code in Storage#tryLock(), there is a possibility that {{
file.getChannel().tryLock();}} returns null if the lock is acquired by some
other process. In that case even though return value is null, a successfull
message confuses.
{code}try {
res = file.getChannel().tryLock();
file.write(jvmName.getBytes(Charsets.UTF_8));
LOG.info(Lock on + lockF + acquired by nodename + jvmName);
} catch(OverlappingFileLockException oe) {{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5669) Storage#tryLock() should check for null before logging successfull message
[
https://issues.apache.org/jira/browse/HDFS-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinay updated HDFS-5669:
Affects Version/s: 2.2.0
Status: Patch Available (was: Open)
Storage#tryLock() should check for null before logging successfull message
--
Key: HDFS-5669
URL: https://issues.apache.org/jira/browse/HDFS-5669
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.2.0
Reporter: Vinay
Assignee: Vinay
Attachments: HDFS-5669.patch
In the following code in Storage#tryLock(), there is a possibility that {{
file.getChannel().tryLock();}} returns null if the lock is acquired by some
other process. In that case even though return value is null, a successfull
message confuses.
{code}try {
res = file.getChannel().tryLock();
file.write(jvmName.getBytes(Charsets.UTF_8));
LOG.info(Lock on + lockF + acquired by nodename + jvmName);
} catch(OverlappingFileLockException oe) {{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5669) Storage#tryLock() should check for null before logging successfull message
[
https://issues.apache.org/jira/browse/HDFS-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinay updated HDFS-5669:
Attachment: HDFS-5669.patch
Storage#tryLock() should check for null before logging successfull message
--
Key: HDFS-5669
URL: https://issues.apache.org/jira/browse/HDFS-5669
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.2.0
Reporter: Vinay
Assignee: Vinay
Attachments: HDFS-5669.patch
In the following code in Storage#tryLock(), there is a possibility that {{
file.getChannel().tryLock();}} returns null if the lock is acquired by some
other process. In that case even though return value is null, a successfull
message confuses.
{code}try {
res = file.getChannel().tryLock();
file.write(jvmName.getBytes(Charsets.UTF_8));
LOG.info(Lock on + lockF + acquired by nodename + jvmName);
} catch(OverlappingFileLockException oe) {{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5669) Storage#tryLock() should check for null before logging successfull message
[
https://issues.apache.org/jira/browse/HDFS-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinay updated HDFS-5669:
Description:
In the following code in Storage#tryLock(), there is a possibility that
{{file.getChannel().tryLock()}} returns null if the lock is acquired by some
other process. In that case even though return value is null, a successfull
message confuses.
{code}try {
res = file.getChannel().tryLock();
file.write(jvmName.getBytes(Charsets.UTF_8));
LOG.info(Lock on + lockF + acquired by nodename + jvmName);
} catch(OverlappingFileLockException oe) {{code}
was:
In the following code in Storage#tryLock(), there is a possibility that {{
file.getChannel().tryLock();}} returns null if the lock is acquired by some
other process. In that case even though return value is null, a successfull
message confuses.
{code}try {
res = file.getChannel().tryLock();
file.write(jvmName.getBytes(Charsets.UTF_8));
LOG.info(Lock on + lockF + acquired by nodename + jvmName);
} catch(OverlappingFileLockException oe) {{code}
Storage#tryLock() should check for null before logging successfull message
--
Key: HDFS-5669
URL: https://issues.apache.org/jira/browse/HDFS-5669
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.2.0
Reporter: Vinay
Assignee: Vinay
Attachments: HDFS-5669.patch
In the following code in Storage#tryLock(), there is a possibility that
{{file.getChannel().tryLock()}} returns null if the lock is acquired by some
other process. In that case even though return value is null, a successfull
message confuses.
{code}try {
res = file.getChannel().tryLock();
file.write(jvmName.getBytes(Charsets.UTF_8));
LOG.info(Lock on + lockF + acquired by nodename + jvmName);
} catch(OverlappingFileLockException oe) {{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-4839) add NativeIO#mkdirs, that provides an error message on failure
[ https://issues.apache.org/jira/browse/HDFS-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848417#comment-13848417 ] Steve Loughran commented on HDFS-4839: -- If this can be done in Java7 then its not worth doing here, unless a goal is to backport behaviour to java6 runtimes. Irrespective of how this is done, it changes the behaviour of local file:// operations when things fail. While this is probably a good thing, it may break things. Perhaps it should just be catch+log for now: logging can only be useful. In which case, this could maybe be implemented with reflection today add NativeIO#mkdirs, that provides an error message on failure -- Key: HDFS-4839 URL: https://issues.apache.org/jira/browse/HDFS-4839 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Priority: Minor It would be nice to have a variant of mkdirs that provided an error message explaining why it failed. This would make it easier to debug certain failing unit tests that rely on mkdir / mkdirs-- the ChecksumFilesystem tests, for example. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5660) When SSL is enabled , the Namenode WEBUI redirects to Infosecport, which could be 0
[ https://issues.apache.org/jira/browse/HDFS-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848423#comment-13848423 ] Benoy Antony commented on HDFS-5660: The bug exists in 2.2 . The bug was introduced by HDFS-5307 which changed the redirectPort to infosecurePort when HTTPS is enabled. But infoSecurePort is valid only when dfs.https.enable is true . if you set only hadoop.ssl.enabled to true, infoSecurePort is 0. If you are in 2,.2, you need this patch to make things work as it used to work. I believe , the bug doesn't exist in trunk by looking at the code. When SSL is enabled , the Namenode WEBUI redirects to Infosecport, which could be 0 --- Key: HDFS-5660 URL: https://issues.apache.org/jira/browse/HDFS-5660 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.2.0 Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HDFS-5660.patch (case 1) When SSL is enabled by setting hadoop.ssl.enabled, SSL will be enabled on the regular port (infoport) on Datanode. (case 2) When SSL on HDFS is enabled by setting dfs.https.enable, SSL will be enabled on a separate port (infoSecurePort) on Datanode. if SSL is enabled , the Namenode always redirects to infoSecurePort. infoSecurePort will be 0 in case 1 above. This breaks the file browsing via web. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5669) Storage#tryLock() should check for null before logging successfull message
[
https://issues.apache.org/jira/browse/HDFS-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848436#comment-13848436
]
Hadoop QA commented on HDFS-5669:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618782/HDFS-5669.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5722//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5722//console
This message is automatically generated.
Storage#tryLock() should check for null before logging successfull message
--
Key: HDFS-5669
URL: https://issues.apache.org/jira/browse/HDFS-5669
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.2.0
Reporter: Vinay
Assignee: Vinay
Attachments: HDFS-5669.patch
In the following code in Storage#tryLock(), there is a possibility that
{{file.getChannel().tryLock()}} returns null if the lock is acquired by some
other process. In that case even though return value is null, a successfull
message confuses.
{code}try {
res = file.getChannel().tryLock();
file.write(jvmName.getBytes(Charsets.UTF_8));
LOG.info(Lock on + lockF + acquired by nodename + jvmName);
} catch(OverlappingFileLockException oe) {{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5661) Browsing FileSystem via web ui, should use datanode's hostname instead of ip address
[
https://issues.apache.org/jira/browse/HDFS-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848440#comment-13848440
]
Haohui Mai commented on HDFS-5661:
--
bq. AuthFilter.java is used only for webhdfs. While accessing JSP files,
AuthenticationFilter is used and AuthenticationFilter does not use
delegationToken.
All meaningful JSP on the datadode server (i.e., tail / browseBlock /
browseDirectory) accesses the HDFS. You cannot proceed without a delegation
token.
If you are able to access it without a DT, this is a security vulnerability and
please file a jira to report it.
bq. Note that the use of IP address while generating redirectURL was introduced
with HDFS-5307. It used to be fqdn before.
It calls {{InetSocketAddress#getCanonicalHostName()}} internally. It is broken
when the machine have multiple DNS names.
Popping up one level, can you please restate what you are trying to achieve?
The old UI is no longer under active development, it may be deprecated or
removed at some point. It may be worthwhile to spend the time of migrating to
the new UI.
Browsing FileSystem via web ui, should use datanode's hostname instead of ip
address
Key: HDFS-5661
URL: https://issues.apache.org/jira/browse/HDFS-5661
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-5661.patch
If authentication is enabled on the web ui, then a cookie is used to keep
track of the authentication information. There is normally a domain
associated with the cookie. Since ip address doesn't have any domain , the
cookie will not be sent by the browser while making http calls with ip
address as the destination server.
This will break browsing files system via web ui , if authentication is
enabled.
Browsing FileSystem via web ui, should use datanode's hostname instead of ip
address.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5660) When SSL is enabled , the Namenode WEBUI redirects to Infosecport, which could be 0
[ https://issues.apache.org/jira/browse/HDFS-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848444#comment-13848444 ] Haohui Mai commented on HDFS-5660: -- Sorry, what I meant was the bug had no longer existed after 2.2. bq. If you are in 2,.2, you need this patch to make things work as it used to work. hadoop.ssl.enabled is introduced in the 2.2 and it is considered broken at the very first day. It will be removed in the next subsequent release so this is a clearly wontfix to me. Popping up one level, even if this bug should be fixed, I don't think it will fit in anywhere. Obviously it won't go into trunk / branch-2, I don't think it'll be picked up by branch-2.3 either, as it intends to contain only critical fixes for 2.2. When SSL is enabled , the Namenode WEBUI redirects to Infosecport, which could be 0 --- Key: HDFS-5660 URL: https://issues.apache.org/jira/browse/HDFS-5660 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.2.0 Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HDFS-5660.patch (case 1) When SSL is enabled by setting hadoop.ssl.enabled, SSL will be enabled on the regular port (infoport) on Datanode. (case 2) When SSL on HDFS is enabled by setting dfs.https.enable, SSL will be enabled on a separate port (infoSecurePort) on Datanode. if SSL is enabled , the Namenode always redirects to infoSecurePort. infoSecurePort will be 0 in case 1 above. This breaks the file browsing via web. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4839) add NativeIO#mkdirs, that provides an error message on failure
[
https://issues.apache.org/jira/browse/HDFS-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848453#comment-13848453
]
Chris Nauroth commented on HDFS-4839:
-
bq. Irrespective of how this is done, it changes the behaviour of local file://
operations when things fail.
From a compatibility perspective, I think this means we'd have to be careful
that the change doesn't start throwing {{IOException}} in cases that used to
return {{false}}. As you said, just logging the information would be a way to
handle this. I think this meets our goal to provide enough information for
diagnosis. (I don't think anyone is proposing that we propagate the native
{{errno}} all the way out to the caller of {{FileSystem#mkdirs}}.)
bq. In which case, this could maybe be implemented with reflection today
Sorry, I didn't follow this part. Could you explain? I've been thinking of
this change as calling C {{mkdir}}, then getting the {{errno}}, and then
pushing that back up to the Java layer for logging. How can reflection help
with this? Is the {{errno}} hidden somewhere in the JDK classes, and you're
proposing that we peek at it with reflection?
Thanks!
add NativeIO#mkdirs, that provides an error message on failure
--
Key: HDFS-4839
URL: https://issues.apache.org/jira/browse/HDFS-4839
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.1.0-beta
Reporter: Colin Patrick McCabe
Priority: Minor
It would be nice to have a variant of mkdirs that provided an error message
explaining why it failed. This would make it easier to debug certain failing
unit tests that rely on mkdir / mkdirs-- the ChecksumFilesystem tests, for
example.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Attachment: HDFS-5454.03.patch
Update patch to add test case.
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Priority: Minor
Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
HDFS-5454.03.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848512#comment-13848512
]
Hadoop QA commented on HDFS-5454:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618793/HDFS-5454.03.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5723//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5723//console
This message is automatically generated.
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Priority: Minor
Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
HDFS-5454.03.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5593) Test cases for split and combined block reports
[ https://issues.apache.org/jira/browse/HDFS-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5593: Resolution: Duplicate Status: Resolved (was: Patch Available) I will include this test along with HDFS-5406, (and include an additional test case for HDFS-5406 in the same new test class). Test cases for split and combined block reports --- Key: HDFS-5593 URL: https://issues.apache.org/jira/browse/HDFS-5593 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: h5593.01.patch, h5593.02.patch Now that we treat the Datanode as a collection of storages, Datanodes can choose to split or combine block reports from individual storages. We need new tests to verify that the NameNode can correctly handle different variations of block reports. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5406) Send incremental block reports for all storages in a single call
[
https://issues.apache.org/jira/browse/HDFS-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5406:
Attachment: h5406.05.patch
Updated patch to add new test case for this fix
({{TestIncrementalBrVariations#testDataNodeDoesNotSplitReports}}) and also
include all tests from HDFS-5593.
Send incremental block reports for all storages in a single call
Key: HDFS-5406
URL: https://issues.apache.org/jira/browse/HDFS-5406
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Attachments: h5406.01.patch, h5406.02.patch, h5406.04.patch,
h5406.05.patch
Per code review feedback from [~szetszwo] on HDFS-5390, we can combine all
incremental block reports in a single {{blockReceivedAndDeleted}} call.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Created] (HDFS-5670) FSPermission check is incorrect
Fengdong Yu created HDFS-5670:
-
Summary: FSPermission check is incorrect
Key: HDFS-5670
URL: https://issues.apache.org/jira/browse/HDFS-5670
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client, namenode
Affects Versions: 2.2.0, 3.0.0
Reporter: Fengdong Yu
Fix For: 3.0.0, 2.3.0
FSPermission check is incorrect after update in the trunk recently.
I submitted MR job using root, but the whole output directory must be owned by
root, otherwise, it throws Exception:
{code}
[root@10 ~]# hadoop fs -ls /
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2013-12-15 10:04 /user
[root@10 ~]#
[root@10 ~]# hadoop fs -ls /user
Found 1 items
drwxr-xr-x - root root 0 2013-12-15 10:04 /user/root
{code}
{code}
[root@10 ~]# hadoop jar airui.jar /input /user/root/
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=root, access=WRITE,
inode=/user:hadoop:supergroup:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:234)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:214)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:161)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5410)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:3236)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:3190)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3174)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:708)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:514)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5406) Send incremental block reports for all storages in a single call
[
https://issues.apache.org/jira/browse/HDFS-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848546#comment-13848546
]
Hadoop QA commented on HDFS-5406:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618796/h5406.05.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5724//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5724//console
This message is automatically generated.
Send incremental block reports for all storages in a single call
Key: HDFS-5406
URL: https://issues.apache.org/jira/browse/HDFS-5406
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Attachments: h5406.01.patch, h5406.02.patch, h5406.04.patch,
h5406.05.patch
Per code review feedback from [~szetszwo] on HDFS-5390, we can combine all
incremental block reports in a single {{blockReceivedAndDeleted}} call.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5664) try to relieve the BlockReaderLocal read() synchronized hotspot
[
https://issues.apache.org/jira/browse/HDFS-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848545#comment-13848545
]
stack commented on HDFS-5664:
-
bq. why HBase use one stream always for one HFile in history？
Opening a new stream requires a trip to the namenode. Doing that for every
get/scan would up latencies significantly (and likely buckle the NN). We have
discussed in the past opening a new stream if a long-running scan is started --
if we recognize a scan as long-running -- so we can exploit any DN
pipelining We also recently changed from seek+read to pread if we
recognize we have more than one scanner going against the same set of files
because it improved our throughput.
bq. A single DFSInputStream is designed to be used by a single thread only.
Thanks Colin. Then we should remove all synchronization if single threaded
only?
Could we save on NN trips if we had added a 'clone' of DFSIS where'd create a
new one passing in an existing one; the new DFSIS would use the block info the
original had already obtained which would be enough to get the new DFSIS off
the ground w/o a trip to the NN?
Thanks
try to relieve the BlockReaderLocal read() synchronized hotspot
---
Key: HDFS-5664
URL: https://issues.apache.org/jira/browse/HDFS-5664
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.0.0, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
Current the BlockReaderLocal's read has a synchronized modifier:
{code}
public synchronized int read(byte[] buf, int off, int len) throws IOException
{
{code}
In a HBase physical read heavy cluster, we observed some hotspots from
dfsclient path, the detail strace trace could be found from:
https://issues.apache.org/jira/browse/HDFS-1605?focusedCommentId=13843241page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13843241
I haven't looked into the detail yet, put some raw ideas here firstly:
1) replace synchronized with try lock with timeout pattern, so could
fail-fast, 2) fallback to non-ssr mode if get a local reader lock failed.
There're two suitable scenario at least to remove this hotspot:
1) Local physical read heavy, e.g. HBase block cache miss ratio is high
2) slow/bad disk.
It would be helpful to achive a lower 99th percentile HBase read latency
somehow.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5663) make the retry time and interval value configurable in openInfo()
[ https://issues.apache.org/jira/browse/HDFS-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848556#comment-13848556 ] Liang Xie commented on HDFS-5663: - The failed case was not caused by this JIRA. more comments? make the retry time and interval value configurable in openInfo() - Key: HDFS-5663 URL: https://issues.apache.org/jira/browse/HDFS-5663 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0, 2.2.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-5663.txt The original idea raised from Michael here: https://issues.apache.org/jira/browse/HDFS-1605?focusedCommentId=13846972page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13846972 It would be better to have a lower interval value especially for online service, like HBase. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5663) make the retry time and interval value configurable in openInfo()
[ https://issues.apache.org/jira/browse/HDFS-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848557#comment-13848557 ] Liang Xie commented on HDFS-5663: - The failed case was not caused by this JIRA. more comments? make the retry time and interval value configurable in openInfo() - Key: HDFS-5663 URL: https://issues.apache.org/jira/browse/HDFS-5663 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0, 2.2.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-5663.txt The original idea raised from Michael here: https://issues.apache.org/jira/browse/HDFS-1605?focusedCommentId=13846972page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13846972 It would be better to have a lower interval value especially for online service, like HBase. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
