[jira] [Commented] (HDFS-1148) Convert FSDataset to ReadWriteLock
[
https://issues.apache.org/jira/browse/HDFS-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654926#comment-14654926
]
Hadoop QA commented on HDFS-1148:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 6s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 12s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 0s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 28s | The applied patch generated 5
new checkstyle issues (total was 122, now 115). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 25s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 39s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 96m 22s | Tests failed in hadoop-hdfs. |
| | | 143m 34s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.namenode.TestAclConfigFlag |
| | hadoop.hdfs.server.namenode.TestListCorruptFileBlocks |
| | hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant |
| | hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd |
| | hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748681/HDFS-1148.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d540374 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11901/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11901/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11901/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11901/console |
This message was automatically generated.
> Convert FSDataset to ReadWriteLock
> --
>
> Key: HDFS-1148
> URL: https://issues.apache.org/jira/browse/HDFS-1148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, performance
>Reporter: Todd Lipcon
>Assignee: Yong Zhang
> Attachments: HDFS-1148.001.patch, hdfs-1148-old.txt,
> hdfs-1148-trunk.txt, patch-HDFS-1148-rel0.20.2.txt
>
>
> In benchmarking HDFS-941 I noticed that for the random read workload, the
> FSDataset lock is highly contended. After converting it to a
> ReentrantReadWriteLock, I saw a ~25% improvement on both latency and
> ops/second.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8858) DU should be re-executed if the target directory exists
Akira AJISAKA created HDFS-8858: --- Summary: DU should be re-executed if the target directory exists Key: HDFS-8858 URL: https://issues.apache.org/jira/browse/HDFS-8858 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Akira AJISAKA Priority: Minor Unix DU command rarely fails when a child file/directory of the target path is being moved or deleted. I'm thinking we should re-try DU if the target path exists, to avoid failure in writing replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8858) DU should be re-executed if the target directory exists
[ https://issues.apache.org/jira/browse/HDFS-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654980#comment-14654980 ] Masatake Iwasaki commented on HDFS-8858: This seems to be related to HDFS-8791. > DU should be re-executed if the target directory exists > --- > > Key: HDFS-8858 > URL: https://issues.apache.org/jira/browse/HDFS-8858 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Akira AJISAKA >Priority: Minor > > Unix DU command rarely fails when a child file/directory of the target path > is being moved or deleted. I'm thinking we should re-try DU if the target > path exists, to avoid failure in writing replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654983#comment-14654983
]
Yi Liu commented on HDFS-8856:
--
+1, thanks Arpit.
I found a nit in {{getCompleteBlocksTotal}}, if you can modify it too, that
would be nice.
{code}
readLock();
numUCBlocks = leaseManager.getNumUnderConstructionBlocks();
{code}
It's better to put {{readLock}} below
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8857) Erasure Coding: Fix ArrayIndexOutOfBoundsException in TestWriteStripedFileWithFailure
[
https://issues.apache.org/jira/browse/HDFS-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14655051#comment-14655051
]
Hadoop QA commented on HDFS-8857:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 5m 58s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 52s | There were no new javac warning
messages. |
| {color:red}-1{color} | release audit | 0m 14s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 39s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 40s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 25s | The patch appears to introduce 5
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 1m 22s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 94m 45s | Tests failed in hadoop-hdfs. |
| | | 116m 33s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.TestRead |
| | hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional |
| | hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd |
| | hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
| | hadoop.hdfs.server.datanode.TestRefreshNamenodes |
| | hadoop.hdfs.TestHdfsAdmin |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN |
| | hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler |
| | hadoop.hdfs.TestClientReportBadBlock |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy |
| | hadoop.hdfs.server.blockmanagement.TestDatanodeManager |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.TestWriteStripedFileWithFailure |
| |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement |
| | hadoop.hdfs.server.namenode.TestNameNodeRpcServer |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade |
| | hadoop.cli.TestErasureCodingCLI |
| | hadoop.hdfs.server.namenode.ha.TestHAFsck |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.namenode.TestHDFSConcat |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.TestRefreshCallQueue |
| | hadoop.hdfs.TestListFilesInDFS |
| | hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold |
| | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
| | hadoop.hdfs.TestMiniDFSCluster |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.security.TestPermissionSymlinks |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica |
| | hadoop.hdfs.TestFileConcurrentReader |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.server.datanode.TestDataNodeExit |
| | hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId |
| | hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA |
| | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.security.TestRefreshUserMappings |
| | hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.TestMetadataVersionOutput |
| | hadoop.hdfs.server.namenode.ha.TestHAMetrics |
| | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.server.namenode.TestAllowFormat |
| | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| | hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.server.namenode.TestDeadDatanode |
| | hadoop.hdfs.crypto.TestHdfsCryptoStreams |
| | hadoop.hdfs.server.blockmanagement.TestAvailableSpaceBlockPlacementPolicy
|
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart |
| | hadoop.hdfs.server.datanode.TestTriggerBlockReport |
|
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14655078#comment-14655078
]
Hadoop QA commented on HDFS-8856:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 10s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 21s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 33s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 36s | The applied patch generated 2
new checkstyle issues (total was 296, now 293). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 12 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 25s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 43s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 166m 27s | Tests failed in hadoop-hdfs. |
| | | 214m 30s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| Timed out tests |
org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication |
| | org.apache.hadoop.hdfs.server.namenode.TestFsck |
| | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748780/HDFS-8856.01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d540374 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11902/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11902/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11902/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11902/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11902/console |
This message was automatically generated.
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8853) Erasure Coding: Provide ECSchema validation when creating ECZone
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8853:
-
Attachment: HDFS-8853-HDFS-7285-01.patch
Attached an initial patch .
Please review.
> Erasure Coding: Provide ECSchema validation when creating ECZone
>
>
> Key: HDFS-8853
> URL: https://issues.apache.org/jira/browse/HDFS-8853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: J.Andreina
> Attachments: HDFS-8853-HDFS-7285-01.patch
>
>
> Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
> doesn't have any validation that the given {{ecSchema}} is available in
> {{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
> then will create the ECZone with {{null}} schema. IMHO we could improve this
> by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8853) Erasure Coding: Provide ECSchema validation when creating ECZone
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8853:
-
Status: Patch Available (was: Open)
> Erasure Coding: Provide ECSchema validation when creating ECZone
>
>
> Key: HDFS-8853
> URL: https://issues.apache.org/jira/browse/HDFS-8853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: J.Andreina
> Attachments: HDFS-8853-HDFS-7285-01.patch
>
>
> Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
> doesn't have any validation that the given {{ecSchema}} is available in
> {{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
> then will create the ECZone with {{null}} schema. IMHO we could improve this
> by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8815) DFS getStoragePolicy implementation using single RPC call
[
https://issues.apache.org/jira/browse/HDFS-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-8815:
-
Attachment: HDFS-8815-004.patch
> DFS getStoragePolicy implementation using single RPC call
> -
>
> Key: HDFS-8815
> URL: https://issues.apache.org/jira/browse/HDFS-8815
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-8815-001.patch, HDFS-8815-002.patch,
> HDFS-8815-003.patch, HDFS-8815-004.patch
>
>
> HADOOP-12161 introduced a new {{FileSystem#getStoragePolicy}} call. The DFS
> implementation of the call requires two RPC calls, the first to fetch the
> storage policy ID and the second to fetch the policy suite to map the policy
> ID to a {{BlockStoragePolicySpi}}.
> Fix the implementation to require a single RPC call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8815) DFS getStoragePolicy implementation using single RPC call
[
https://issues.apache.org/jira/browse/HDFS-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14655135#comment-14655135
]
Surendra Singh Lilhore commented on HDFS-8815:
--
Thanks [~vinayrpet] for review..
Attached updated patch, Please review...
> DFS getStoragePolicy implementation using single RPC call
> -
>
> Key: HDFS-8815
> URL: https://issues.apache.org/jira/browse/HDFS-8815
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-8815-001.patch, HDFS-8815-002.patch,
> HDFS-8815-003.patch, HDFS-8815-004.patch
>
>
> HADOOP-12161 introduced a new {{FileSystem#getStoragePolicy}} call. The DFS
> implementation of the call requires two RPC calls, the first to fetch the
> storage policy ID and the second to fetch the policy suite to map the policy
> ID to a {{BlockStoragePolicySpi}}.
> Fix the implementation to require a single RPC call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8802) dfs.checksum.type is not described in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-8802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14655137#comment-14655137 ] Gururaj Shetty commented on HDFS-8802: -- Hi [~ozawa] Kindly review and let me know for any update. Thanks > dfs.checksum.type is not described in hdfs-default.xml > -- > > Key: HDFS-8802 > URL: https://issues.apache.org/jira/browse/HDFS-8802 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 >Reporter: Tsuyoshi Ozawa >Assignee: Gururaj Shetty > Attachments: HDFS-8802.patch, HDFS-8802_01.patch, HDFS-8802_02.patch > > > It's a good timing to check other configurations about hdfs-default.xml here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8701) WebHDFS write commands fail when running multiple DataNodes on one machine
[
https://issues.apache.org/jira/browse/HDFS-8701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14655139#comment-14655139
]
Jagadesh Kiran N commented on HDFS-8701:
hi [~erwaman] , i tried with 3 datanodes in same machine, it is working fine.
Can you please let me know if any specific scenario where put or touch fails?
> WebHDFS write commands fail when running multiple DataNodes on one machine
> --
>
> Key: HDFS-8701
> URL: https://issues.apache.org/jira/browse/HDFS-8701
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Jagadesh Kiran N
>
> For testing purposes, we are running multiple DataNodes per machine. {{hadoop
> fs}} commands work fine when using the {{hdfs://}} protocol, but when using
> {{webhdfs://}}, any command that writes to HDFS (e.g.: {{-put}} or
> {{-touchz}}) fails:
> {code}
> $ hadoop fs -put test.txt webhdfs://:/user/foo
> put: -dn-4
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8844) TestHDFSCLI does not cleanup the test directory
[
https://issues.apache.org/jira/browse/HDFS-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14655172#comment-14655172
]
Hudson commented on HDFS-8844:
--
ABORTED: Integrated in Hadoop-Yarn-trunk #1008 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1008/])
HDFS-8844. TestHDFSCLI does not cleanup the test directory (Masatake Iwasaki
via Colin P. McCabe) (cmccabe: rev c95993cbaf51e2925ea9b1b95cf4f0d879e66489)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
> TestHDFSCLI does not cleanup the test directory
> ---
>
> Key: HDFS-8844
> URL: https://issues.apache.org/jira/browse/HDFS-8844
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8844.001.patch
>
>
> If TestHDFSCLI is executed twice without {{mvn clean}}, the second try fails.
> Here are the failing test cases:
> {noformat}
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(231)) - Failing tests:
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(232)) - --
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 226: get: getting non
> existent(absolute path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 227: get: getting non existent
> file(relative path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 228: get: Test for hdfs:// path -
> getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 229: get: Test for Namenode's path
> - getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 250: copyToLocal: non existent
> relative path
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 251: copyToLocal: non existent
> absolute path
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 252: copyToLocal: Test for hdfs://
> path - non existent file/directory
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 253: copyToLocal: Test for
> Namenode's path - non existent file/directory
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8844) TestHDFSCLI does not cleanup the test directory
[
https://issues.apache.org/jira/browse/HDFS-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14655175#comment-14655175
]
Hudson commented on HDFS-8844:
--
ABORTED: Integrated in Hadoop-Yarn-trunk-Java8 #278 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/278/])
HDFS-8844. TestHDFSCLI does not cleanup the test directory (Masatake Iwasaki
via Colin P. McCabe) (cmccabe: rev c95993cbaf51e2925ea9b1b95cf4f0d879e66489)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
> TestHDFSCLI does not cleanup the test directory
> ---
>
> Key: HDFS-8844
> URL: https://issues.apache.org/jira/browse/HDFS-8844
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8844.001.patch
>
>
> If TestHDFSCLI is executed twice without {{mvn clean}}, the second try fails.
> Here are the failing test cases:
> {noformat}
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(231)) - Failing tests:
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(232)) - --
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 226: get: getting non
> existent(absolute path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 227: get: getting non existent
> file(relative path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 228: get: Test for hdfs:// path -
> getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 229: get: Test for Namenode's path
> - getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 250: copyToLocal: non existent
> relative path
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 251: copyToLocal: non existent
> absolute path
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 252: copyToLocal: Test for hdfs://
> path - non existent file/directory
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 253: copyToLocal: Test for
> Namenode's path - non existent file/directory
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8854: Summary: Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs (was: Erasure coding: Move cellSize inside ECSchema) > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Walter Su >Assignee: Walter Su > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8859) Improve DataNode (ReplicaMap) memory footprint to save about 45%
Yi Liu created HDFS-8859:
Summary: Improve DataNode (ReplicaMap) memory footprint to save
about 45%
Key: HDFS-8859
URL: https://issues.apache.org/jira/browse/HDFS-8859
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Reporter: Yi Liu
Assignee: Yi Liu
By using following approach we can save about *45%* memory footprint for each
block replica in DataNode memory (This JIRA only talks about *ReplicaMap* in
DataNode), the details are:
In ReplicaMap,
{code}
private final Map> map =
new HashMap>();
{code}
Currently we use a HashMap {{Map}} to store the replicas in
memory. The key is block id of the block replica which is already included in
{{ReplicaInfo}}, so this memory can be saved. Also HashMap Entry has a object
overhead. We can implement a lightweight Set which is similar to
{{LightWeightGSet}}, but not a fixed size ({{LightWeightGSet}} uses fix size
for the entries array, usually it's a big value, an example is {{BlocksMap}},
this can avoid full gc since no need to resize), also we should be able to get
Element through key.
Following is comparison of memory footprint If we implement a lightweight set
as described:
We can save:
{noformat}
SIZE (bytes) ITEM
20The Key: Long (12 bytes object overhead + 8 bytes
long)
12HashMap Entry object overhead
4 reference to the key in Entry
4 reference to the value in Entry
4 hash in Entry
{noformat}
Total: -44 bytes
We need to add:
{noformat}
SIZE (bytes) ITEM
4 a reference to next element in ReplicaInfo
{noformat}
Total: +4 bytes
So totally we can save 40bytes for each block replica
And currently one finalized replica needs around 46 bytes (notice: we ignore
memory alignment here).
We can save 1 - (4 + 46) / (44 + 46) = *45%* memory for each block replica in
DataNode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8854:
Attachment: HDFS-8854.00.patch
bq. Maybe we can keep ECSchema class in COMMON, and refactor HDFS codes to use
ErasureCodingPolicy instead?
Yeah, that's better. I updated the jira description.
Upload initial patch. I'll revisit the indent and javadoc later.
The patch does:
1. add {{ECPolicy}} in hadoop-hdfs. {{ECSchema}} is left and only used in
hadoop-common(codec).
2. Rename {{SchemaLoader}} to {{ECPolicyLoader}},
{{ErasureCodingSchemaManager}} to {{ErasureCodingPolicyManager}}.
3. {{ECPolicy}} is loaded from file. The file definition is :
{code}
RS-6-3-64k
6
3
RS
65536
RS-10-4-128k
10
4
RS
131072
{code}
4. {{ECSchema}} is not loaded from file anymore. {{ECSchema}} *is got from*
{{ECPolicy}}.
5. {{ECPolicy}} is cached instead of {{ECSchema}}.
6. update CLI. cellSize is no longer accepted.
7. pass around {{ECPolicy}} instead of {{ECSchema}} + {{cellSize}}. Update
proto, protocol.
hints for review: The patch is huge because of #7. You can focus on #1~#6.
> Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
> --
>
> Key: HDFS-8854
> URL: https://issues.apache.org/jira/browse/HDFS-8854
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8854.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8844) TestHDFSCLI does not cleanup the test directory
[
https://issues.apache.org/jira/browse/HDFS-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658192#comment-14658192
]
Hudson commented on HDFS-8844:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #267 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/267/])
HDFS-8844. TestHDFSCLI does not cleanup the test directory (Masatake Iwasaki
via Colin P. McCabe) (cmccabe: rev c95993cbaf51e2925ea9b1b95cf4f0d879e66489)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
> TestHDFSCLI does not cleanup the test directory
> ---
>
> Key: HDFS-8844
> URL: https://issues.apache.org/jira/browse/HDFS-8844
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8844.001.patch
>
>
> If TestHDFSCLI is executed twice without {{mvn clean}}, the second try fails.
> Here are the failing test cases:
> {noformat}
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(231)) - Failing tests:
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(232)) - --
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 226: get: getting non
> existent(absolute path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 227: get: getting non existent
> file(relative path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 228: get: Test for hdfs:// path -
> getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 229: get: Test for Namenode's path
> - getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 250: copyToLocal: non existent
> relative path
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 251: copyToLocal: non existent
> absolute path
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 252: copyToLocal: Test for hdfs://
> path - non existent file/directory
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 253: copyToLocal: Test for
> Namenode's path - non existent file/directory
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8844) TestHDFSCLI does not cleanup the test directory
[
https://issues.apache.org/jira/browse/HDFS-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658271#comment-14658271
]
Hudson commented on HDFS-8844:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2205 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2205/])
HDFS-8844. TestHDFSCLI does not cleanup the test directory (Masatake Iwasaki
via Colin P. McCabe) (cmccabe: rev c95993cbaf51e2925ea9b1b95cf4f0d879e66489)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
> TestHDFSCLI does not cleanup the test directory
> ---
>
> Key: HDFS-8844
> URL: https://issues.apache.org/jira/browse/HDFS-8844
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8844.001.patch
>
>
> If TestHDFSCLI is executed twice without {{mvn clean}}, the second try fails.
> Here are the failing test cases:
> {noformat}
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(231)) - Failing tests:
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(232)) - --
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 226: get: getting non
> existent(absolute path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 227: get: getting non existent
> file(relative path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 228: get: Test for hdfs:// path -
> getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 229: get: Test for Namenode's path
> - getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 250: copyToLocal: non existent
> relative path
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 251: copyToLocal: non existent
> absolute path
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 252: copyToLocal: Test for hdfs://
> path - non existent file/directory
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 253: copyToLocal: Test for
> Namenode's path - non existent file/directory
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8855) Webhdfs client leaks active NameNode connections
[ https://issues.apache.org/jira/browse/HDFS-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658298#comment-14658298 ] Bob Hansen commented on HDFS-8855: -- This has the same description as HDFS-7597, although the patch is in the UGI space, which surprises me. It speaks of the RPC layer caching the NN connection but that cache not being re-used because the UGI is created anew for each webhdfs request. > Webhdfs client leaks active NameNode connections > > > Key: HDFS-8855 > URL: https://issues.apache.org/jira/browse/HDFS-8855 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs > Environment: HDP 2.2 >Reporter: Bob Hansen >Assignee: Xiaobing Zhou > > The attached script simulates a process opening ~50 files via webhdfs and > performing random reads. Note that there are at most 50 concurrent reads, > and all webhdfs sessions are kept open. Each read is ~64k at a random > position. > The script periodically (once per second) shells into the NameNode and > produces a summary of the socket states. For my test cluster with 5 nodes, > it took ~30 seconds for the NameNode to have ~25000 active connections and > fails. > It appears that each request to the webhdfs client is opening a new > connection to the NameNode and keeping it open after the request is complete. > If the process continues to run, eventually (~30-60 seconds), all of the > open connections are closed and the NameNode recovers. > This smells like SoftReference reaping. Are we using SoftReferences in the > webhdfs client to cache NameNode connections but never re-using them? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8844) TestHDFSCLI does not cleanup the test directory
[
https://issues.apache.org/jira/browse/HDFS-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658355#comment-14658355
]
Hudson commented on HDFS-8844:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #275 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/275/])
HDFS-8844. TestHDFSCLI does not cleanup the test directory (Masatake Iwasaki
via Colin P. McCabe) (cmccabe: rev c95993cbaf51e2925ea9b1b95cf4f0d879e66489)
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestHDFSCLI does not cleanup the test directory
> ---
>
> Key: HDFS-8844
> URL: https://issues.apache.org/jira/browse/HDFS-8844
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8844.001.patch
>
>
> If TestHDFSCLI is executed twice without {{mvn clean}}, the second try fails.
> Here are the failing test cases:
> {noformat}
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(231)) - Failing tests:
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(232)) - --
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 226: get: getting non
> existent(absolute path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 227: get: getting non existent
> file(relative path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 228: get: Test for hdfs:// path -
> getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 229: get: Test for Namenode's path
> - getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 250: copyToLocal: non existent
> relative path
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 251: copyToLocal: non existent
> absolute path
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 252: copyToLocal: Test for hdfs://
> path - non existent file/directory
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 253: copyToLocal: Test for
> Namenode's path - non existent file/directory
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8844) TestHDFSCLI does not cleanup the test directory
[
https://issues.apache.org/jira/browse/HDFS-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658391#comment-14658391
]
Hudson commented on HDFS-8844:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2224 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2224/])
HDFS-8844. TestHDFSCLI does not cleanup the test directory (Masatake Iwasaki
via Colin P. McCabe) (cmccabe: rev c95993cbaf51e2925ea9b1b95cf4f0d879e66489)
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestHDFSCLI does not cleanup the test directory
> ---
>
> Key: HDFS-8844
> URL: https://issues.apache.org/jira/browse/HDFS-8844
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8844.001.patch
>
>
> If TestHDFSCLI is executed twice without {{mvn clean}}, the second try fails.
> Here are the failing test cases:
> {noformat}
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(231)) - Failing tests:
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(232)) - --
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 226: get: getting non
> existent(absolute path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 227: get: getting non existent
> file(relative path)
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 228: get: Test for hdfs:// path -
> getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 229: get: Test for Namenode's path
> - getting non existent
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 250: copyToLocal: non existent
> relative path
> 2015-07-31 21:35:17,654 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 251: copyToLocal: non existent
> absolute path
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 252: copyToLocal: Test for hdfs://
> path - non existent file/directory
> 2015-07-31 21:35:17,655 [main] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(238)) - 253: copyToLocal: Test for
> Namenode's path - non existent file/directory
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8792) Improve BlockManager#postponedMisreplicatedBlocks and BlockManager#excessReplicateMap
[
https://issues.apache.org/jira/browse/HDFS-8792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658477#comment-14658477
]
Colin Patrick McCabe commented on HDFS-8792:
{code}
530 /** The starting modification for fail-fast. */
{code}
This JavaDoc comment is stale. It seems like it should be "the current
modification epoch" or similar.
{code}
public final Map> excessReplicateMap
=
-new TreeMap<>();
+new HashMap<>();
{code}
Can you put this in a separate JIRA? It's less obvious to me that this is a
win. Java {{HashMaps}} don't ever shrink, whereas {{TreeMap}} uses less memory
when elements are removed.
> Improve BlockManager#postponedMisreplicatedBlocks and
> BlockManager#excessReplicateMap
> -
>
> Key: HDFS-8792
> URL: https://issues.apache.org/jira/browse/HDFS-8792
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8792.001.patch, HDFS-8792.002.patch
>
>
> {{LightWeightHashSet}} requires fewer memory than java hashset.
> Furthermore, for {{excessReplicateMap}}, we can use {{HashMap}} instead of
> {{TreeMap}} instead, since no need to sort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8854: Affects Version/s: HDFS-7285 Target Version/s: HDFS-7285 Status: Patch Available (was: Open) Thanks Walter for the work! Submitting patch to trigger Jenkins. > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Walter Su > Attachments: HDFS-8854.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8854: Attachment: HDFS-8854-HDFS-7285.00.patch Submitting duplicate patch, renamed for Jenkins. > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Walter Su > Attachments: HDFS-8854-HDFS-7285.00.patch, HDFS-8854.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8750) FIleSystem does not honor Configuration.getClassLoader() while loading FileSystem implementations
[ https://issues.apache.org/jira/browse/HDFS-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658513#comment-14658513 ] Colin Patrick McCabe commented on HDFS-8750: Is this a compatible change? It seems like changing the classloader that is in use might break some people who were relying on the existing one being used. > FIleSystem does not honor Configuration.getClassLoader() while loading > FileSystem implementations > - > > Key: HDFS-8750 > URL: https://issues.apache.org/jira/browse/HDFS-8750 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs, HDFS >Reporter: Himanshu >Assignee: Himanshu > Attachments: HDFS-8750.001.patch, HDFS-8750.002.patch > > > In FileSystem.loadFileSystems(), at > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L2652 > a "scheme" -> "FileSystem implementation" map is created from the jars > available on classpath. It uses Thread.currentThread().getClassLoader() via > ServiceLoader.load(FileSystem.class) > Instead, loadFileSystems() should take Configuration as an argument and > should first check if a classloader is configured in > configuration.getClassLoader(), if yes then > ServiceLoader.load(FileSystem.class, configuration.getClassLoader()) should > be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658515#comment-14658515 ] Zhe Zhang commented on HDFS-8854: - Took a quite glance, overall structure looks pretty good. Will post a detailed review later today. Great work [~walter.k.su]! > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Walter Su > Attachments: HDFS-8854-HDFS-7285.00.patch, HDFS-8854.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658558#comment-14658558
]
Haohui Mai commented on HDFS-8856:
--
Is it possible to return {{leasesById.size()}} instead?
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8815) DFS getStoragePolicy implementation using single RPC call
[
https://issues.apache.org/jira/browse/HDFS-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658561#comment-14658561
]
Arpit Agarwal commented on HDFS-8815:
-
+1 the v004 patch LGTM.
It is good to avoid {{required}} for new protobuf messages even though we have
used it in the past. {{required}} fields make it hard to evolve the protocol in
a compatible way. You could consider setting the fields to {{optional}} if
you'd like and add an explicit check to make sure they are present. But I'd be
okay with committing it as it is.
Thanks for fixing this for 2.8 [~surendrasingh] and for the reviews
[~vinayrpet].
> DFS getStoragePolicy implementation using single RPC call
> -
>
> Key: HDFS-8815
> URL: https://issues.apache.org/jira/browse/HDFS-8815
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-8815-001.patch, HDFS-8815-002.patch,
> HDFS-8815-003.patch, HDFS-8815-004.patch
>
>
> HADOOP-12161 introduced a new {{FileSystem#getStoragePolicy}} call. The DFS
> implementation of the call requires two RPC calls, the first to fetch the
> storage policy ID and the second to fetch the policy suite to map the policy
> ID to a {{BlockStoragePolicySpi}}.
> Fix the implementation to require a single RPC call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7597) DNs should not open new NN connections when webhdfs clients seek
[ https://issues.apache.org/jira/browse/HDFS-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-7597: - Attachment: HDFS-7597.diff Refactored the diffs to move changes over to hadoop-client project > DNs should not open new NN connections when webhdfs clients seek > > > Key: HDFS-7597 > URL: https://issues.apache.org/jira/browse/HDFS-7597 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Labels: BB2015-05-TBR > Attachments: HDFS-7597.diff, HDFS-7597.patch, HDFS-7597.patch > > > Webhdfs seeks involve closing the current connection, and reissuing a new > open request with the new offset. The RPC layer caches connections so the DN > keeps a lingering connection open to the NN. Connection caching is in part > based on UGI. Although the client used the same token for the new offset > request, the UGI is different which forces the DN to open another unnecessary > connection to the NN. > A job that performs many seeks will easily crash the NN due to fd exhaustion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7597) DNs should not open new NN connections when webhdfs clients seek
[ https://issues.apache.org/jira/browse/HDFS-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-7597: - Attachment: (was: HDFS-7597.diff) > DNs should not open new NN connections when webhdfs clients seek > > > Key: HDFS-7597 > URL: https://issues.apache.org/jira/browse/HDFS-7597 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Labels: BB2015-05-TBR > Attachments: HDFS-7597.patch, HDFS-7597.patch, HDFS-7597.patch > > > Webhdfs seeks involve closing the current connection, and reissuing a new > open request with the new offset. The RPC layer caches connections so the DN > keeps a lingering connection open to the NN. Connection caching is in part > based on UGI. Although the client used the same token for the new offset > request, the UGI is different which forces the DN to open another unnecessary > connection to the NN. > A job that performs many seeks will easily crash the NN due to fd exhaustion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7597) DNs should not open new NN connections when webhdfs clients seek
[ https://issues.apache.org/jira/browse/HDFS-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-7597: - Attachment: HDFS-7597.patch > DNs should not open new NN connections when webhdfs clients seek > > > Key: HDFS-7597 > URL: https://issues.apache.org/jira/browse/HDFS-7597 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Labels: BB2015-05-TBR > Attachments: HDFS-7597.patch, HDFS-7597.patch, HDFS-7597.patch > > > Webhdfs seeks involve closing the current connection, and reissuing a new > open request with the new offset. The RPC layer caches connections so the DN > keeps a lingering connection open to the NN. Connection caching is in part > based on UGI. Although the client used the same token for the new offset > request, the UGI is different which forces the DN to open another unnecessary > connection to the NN. > A job that performs many seeks will easily crash the NN due to fd exhaustion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-8856:
Attachment: HDFS-8856.02.patch
Thanks for the reviews. v2 patch attached.
bq. Is it possible to return leasesById.size() instead?
Fixed.
bq. I found a nit in getCompleteBlocksTotal, if you can modify it too, that
would be nice.
Looks risky, the lock ensures consistent state for each INodeFile.
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658614#comment-14658614
]
Jing Zhao commented on HDFS-8833:
-
Thanks for the discussion guys. Some thoughts here:
One thing I learnt from the storage policy is that using several bits in the
INodeFile's header to represent some policy can always become a big limitation
in the end. When we allow users to define/add new schemas, they always want to
use/define more and more policies (considering we also want to push the cell
size into the policy). How to limit the total number of policies, whether to
allow them to modify or delete an existing policy when the limit is hit, and
how to change the INodeFile layout to support more policies, will all become
challenges for us (HDFS-7076 is an example).
If the main issue here is rename, I'd like to associate the ec policy to a file
only during the rename (instead of during the file creation), i.e., when this
single file (not its ancestral directory) is moved to another directory with a
different ec setting. Also, instead of recording the policy in the file header,
I think we should only use xatrr. Considering most file renames happen in the
same directory, these xattr will not cost much memory.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8838: -- Attachment: HDFS-8838-HDFS-7285-000.patch > Tolerate datanode failures in DFSStripedOutputStream when the data length is > small > -- > > Key: HDFS-8838 > URL: https://issues.apache.org/jira/browse/HDFS-8838 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-8838-HDFS-7285-000.patch, h8838_20150729.patch, > h8838_20150731-HDFS-7285.patch, h8838_20150731.log, h8838_20150731.patch, > h8838_20150804-HDFS-7285.patch, h8838_20150804.patch > > > Currently, DFSStripedOutputStream cannot tolerate datanode failures when the > data length is small. We fix the bugs here and add more tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8838: -- Attachment: (was: h8838_20150804.patch) > Tolerate datanode failures in DFSStripedOutputStream when the data length is > small > -- > > Key: HDFS-8838 > URL: https://issues.apache.org/jira/browse/HDFS-8838 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-8838-HDFS-7285-000.patch, h8838_20150729.patch, > h8838_20150731-HDFS-7285.patch, h8838_20150731.log, h8838_20150731.patch, > h8838_20150804-HDFS-7285.patch > > > Currently, DFSStripedOutputStream cannot tolerate datanode failures when the > data length is small. We fix the bugs here and add more tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658652#comment-14658652 ] Chang Li commented on HDFS-6407: [~wheat9] thanks for the patch! The code looks good. One preference I have is to add Non DFS back to column. Now it's in pop up and we can't sort that information. We rely on that data sometimes. > new namenode UI, lost ability to sort columns in datanode tab > - > > Key: HDFS-6407 > URL: https://issues.apache.org/jira/browse/HDFS-6407 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Nathan Roberts >Assignee: Haohui Mai >Priority: Critical > Labels: BB2015-05-TBR > Attachments: 002-datanodes-sorted-capacityUsed.png, > 002-datanodes.png, 002-filebrowser.png, 002-snapshots.png, > HDFS-6407-002.patch, HDFS-6407-003.patch, HDFS-6407.008.patch, > HDFS-6407.009.patch, HDFS-6407.010.patch, HDFS-6407.4.patch, > HDFS-6407.5.patch, HDFS-6407.6.patch, HDFS-6407.7.patch, HDFS-6407.patch, > browse_directory.png, datanodes.png, snapshots.png, sorting 2.png, sorting > table.png > > > old ui supported clicking on column header to sort on that column. The new ui > seems to have dropped this very useful feature. > There are a few tables in the Namenode UI to display datanodes information, > directory listings and snapshots. > When there are many items in the tables, it is useful to have ability to sort > on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658662#comment-14658662
]
Zhe Zhang commented on HDFS-8833:
-
Thanks Jing for sharing the thoughts!
I agree that a pure XAttr-based solution is simpler, cleaner, and more
scalable. We should probably implement that at this stage and pursue memory
saving ideas as follow-on. [~vinayrpet] / [~walter.k.su] Does the plan sound OK?
[~jingzhao] In the long term, do you think the [hybrid approach |
https://issues.apache.org/jira/browse/HDFS-8833?focusedCommentId=14647996&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14647996]
that Andrew proposed addresses the policy-scalability issue?
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8643) Add snapshot names list to SnapshottableDirectoryStatus
[
https://issues.apache.org/jira/browse/HDFS-8643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-8643:
---
Attachment: HDFS-8643-01.patch
> Add snapshot names list to SnapshottableDirectoryStatus
> ---
>
> Key: HDFS-8643
> URL: https://issues.apache.org/jira/browse/HDFS-8643
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8643-00.patch, HDFS-8643-01.patch
>
>
> The idea of this jira to enhance {{SnapshottableDirectoryStatus}} by adding
> {{snapshotNames}} attribute into it, presently it has the {{snapshotNumber}}.
> IMHO this would help the users to get the list of snapshot names created.
> Also, the snapshot names can be used while renaming or deleting the snapshots.
> {code}
> org.apache.hadoop.hdfs.protocol.SnapshottableDirectoryStatus.java
> /**
>* @return Snapshot names for the directory.
>*/
> public List getSnapshotNames() {
> return snapshotNames;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8643) Add snapshot names list to SnapshottableDirectoryStatus
[
https://issues.apache.org/jira/browse/HDFS-8643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658710#comment-14658710
]
Rakesh R commented on HDFS-8643:
Thanks [~xyao] for the reviews and the pointer to {{lsSnapshottableDir}} cmd.
I've tried to incorporate the changes here and attached another patch.
> Add snapshot names list to SnapshottableDirectoryStatus
> ---
>
> Key: HDFS-8643
> URL: https://issues.apache.org/jira/browse/HDFS-8643
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8643-00.patch, HDFS-8643-01.patch
>
>
> The idea of this jira to enhance {{SnapshottableDirectoryStatus}} by adding
> {{snapshotNames}} attribute into it, presently it has the {{snapshotNumber}}.
> IMHO this would help the users to get the list of snapshot names created.
> Also, the snapshot names can be used while renaming or deleting the snapshots.
> {code}
> org.apache.hadoop.hdfs.protocol.SnapshottableDirectoryStatus.java
> /**
>* @return Snapshot names for the directory.
>*/
> public List getSnapshotNames() {
> return snapshotNames;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8550) Erasure Coding: Fix FindBugs Multithreaded correctness Warning
[
https://issues.apache.org/jira/browse/HDFS-8550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658711#comment-14658711
]
Rakesh R commented on HDFS-8550:
Thanks for sharing the link, I could see all the warnings now. Let me try to
find a better approach to fix this.
> Erasure Coding: Fix FindBugs Multithreaded correctness Warning
> --
>
> Key: HDFS-8550
> URL: https://issues.apache.org/jira/browse/HDFS-8550
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>
> Findbug warning:- Inconsistent synchronization of
> org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 89% of time
> {code}
> Bug type IS2_INCONSISTENT_SYNC (click for details)
> In class org.apache.hadoop.hdfs.DFSOutputStream
> Field org.apache.hadoop.hdfs.DFSOutputStream.streamer
> Synchronized 89% of the time
> Unsynchronized access at DFSOutputStream.java:[line 146]
> Unsynchronized access at DFSOutputStream.java:[line 859]
> Unsynchronized access at DFSOutputStream.java:[line 627]
> Unsynchronized access at DFSOutputStream.java:[line 630]
> Unsynchronized access at DFSOutputStream.java:[line 640]
> Unsynchronized access at DFSOutputStream.java:[line 342]
> Unsynchronized access at DFSOutputStream.java:[line 744]
> Unsynchronized access at DFSOutputStream.java:[line 903]
> Synchronized access at DFSOutputStream.java:[line 737]
> Synchronized access at DFSOutputStream.java:[line 913]
> Synchronized access at DFSOutputStream.java:[line 726]
> Synchronized access at DFSOutputStream.java:[line 756]
> Synchronized access at DFSOutputStream.java:[line 762]
> Synchronized access at DFSOutputStream.java:[line 757]
> Synchronized access at DFSOutputStream.java:[line 758]
> Synchronized access at DFSOutputStream.java:[line 762]
> Synchronized access at DFSOutputStream.java:[line 483]
> Synchronized access at DFSOutputStream.java:[line 486]
> Synchronized access at DFSOutputStream.java:[line 717]
> Synchronized access at DFSOutputStream.java:[line 719]
> Synchronized access at DFSOutputStream.java:[line 722]
> Synchronized access at DFSOutputStream.java:[line 408]
> Synchronized access at DFSOutputStream.java:[line 408]
> Synchronized access at DFSOutputStream.java:[line 423]
> Synchronized access at DFSOutputStream.java:[line 426]
> Synchronized access at DFSOutputStream.java:[line 411]
> Synchronized access at DFSOutputStream.java:[line 452]
> Synchronized access at DFSOutputStream.java:[line 452]
> Synchronized access at DFSOutputStream.java:[line 439]
> Synchronized access at DFSOutputStream.java:[line 439]
> Synchronized access at DFSOutputStream.java:[line 439]
> Synchronized access at DFSOutputStream.java:[line 670]
> Synchronized access at DFSOutputStream.java:[line 580]
> Synchronized access at DFSOutputStream.java:[line 574]
> Synchronized access at DFSOutputStream.java:[line 592]
> Synchronized access at DFSOutputStream.java:[line 583]
> Synchronized access at DFSOutputStream.java:[line 581]
> Synchronized access at DFSOutputStream.java:[line 621]
> Synchronized access at DFSOutputStream.java:[line 609]
> Synchronized access at DFSOutputStream.java:[line 621]
> Synchronized access at DFSOutputStream.java:[line 597]
> Synchronized access at DFSOutputStream.java:[line 612]
> Synchronized access at DFSOutputStream.java:[line 597]
> Synchronized access at DFSOutputStream.java:[line 588]
> Synchronized access at DFSOutputStream.java:[line 624]
> Synchronized access at DFSOutputStream.java:[line 612]
> Synchronized access at DFSOutputStream.java:[line 588]
> Synchronized access at DFSOutputStream.java:[line 632]
> Synchronized access at DFSOutputStream.java:[line 632]
> Synchronized access at DFSOutputStream.java:[line 616]
> Synchronized access at DFSOutputStream.java:[line 633]
> Synchronized access at DFSOutputStream.java:[line 657]
> Synchronized access at DFSOutputStream.java:[line 658]
> Synchronized access at DFSOutputStream.java:[line 695]
> Synchronized access at DFSOutputStream.java:[line 698]
> Synchronized access at DFSOutputStream.java:[line 784]
> Synchronized access at DFSOutputStream.java:[line 795]
> Synchronized access at DFSOutputStream.java:[line 801]
> Synchronized access at DFSOutputStream.java:[line 155]
> Synchronized access at DFSOutputStream.java:[line 158]
> Synchronized access at DFSOutputStream.java:[line 433]
> Synchronized access at DFSOutputStream.java:[line 886]
> Synchronized access at DFSOutputStream.java:[line 463]
> Synchronized access at DFSOutputStream.java:[line 469]
> Synchronized access at DFSOutputStream.java:[line 463]
> Synchronized access at DFSOutputStream.java:[line 470]
> Synchronized access at DFSOutputStream.java:[line 465]
> Synchronized access at DFSOutputStream.java:[line 749]
> Synchronized access at DFSStripedOutputStream.java:[line
[jira] [Commented] (HDFS-8486) DN startup may cause severe data loss
[ https://issues.apache.org/jira/browse/HDFS-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658729#comment-14658729 ] Xiaoyu Yao commented on HDFS-8486: -- Thanks Arpit. The branch-2.6 patch LGTM, +1. > DN startup may cause severe data loss > - > > Key: HDFS-8486 > URL: https://issues.apache.org/jira/browse/HDFS-8486 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.1, 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.7.1 > > Attachments: HDFS-8486-branch-2.6.02.patch, > HDFS-8486-branch-2.6.patch, HDFS-8486.patch, HDFS-8486.patch > > > A race condition between block pool initialization and the directory scanner > may cause a mass deletion of blocks in multiple storages. > If block pool initialization finds a block on disk that is already in the > replica map, it deletes one of the blocks based on size, GS, etc. > Unfortunately it _always_ deletes one of the blocks even if identical, thus > the replica map _must_ be empty when the pool is initialized. > The directory scanner starts at a random time within its periodic interval > (default 6h). If the scanner starts very early it races to populate the > replica map, causing the block pool init to erroneously delete blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658770#comment-14658770
]
Xiaoyu Yao commented on HDFS-8856:
--
Patch v2 looks good to me. +1
One nit: you can use a timeout rule that applies to all cases instead of adding
them individually.
{code}
@Rule
publicTimeout timeout=new Timeout(30);
{code}
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658785#comment-14658785
]
Jing Zhao commented on HDFS-8833:
-
Yes. The hybrid approach is actually the same way we planned for storage policy
(although we're still in the first stage now :( ). To me the second step, i.e.
to allow users to define new EC policies can be left to the next stage of the
EC work, in this way we do not need to make big changes before merging EC
branch to trunk.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658799#comment-14658799
]
Haohui Mai commented on HDFS-8856:
--
{code}
+private Collection getFiles() {
+ return Collections.unmodifiableCollection(files);
+}
{code}
Just wondering whether it is necessary?
{code}
- @Test (timeout=1000)
+ @Test (timeout=30)
{code}
Are you seeing timeouts? I would be surprised as the tests operate on mocked
objects and should be really quick.
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8643) Add snapshot names list to SnapshottableDirectoryStatus
[
https://issues.apache.org/jira/browse/HDFS-8643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658800#comment-14658800
]
Jing Zhao commented on HDFS-8643:
-
Instead of directly adding name list to SnapshottableDirectoryStatus, I think
it may be better to directly enhance the {{lsSnapshottableDir}} cmd by making
multiple RPC calls (i.e., do "ls xxx/.snapshot" for each snapshottable dir).
Note that we have RPC packet limitation and a large number of snapshot list or
some long snapshot names can cause issue.
> Add snapshot names list to SnapshottableDirectoryStatus
> ---
>
> Key: HDFS-8643
> URL: https://issues.apache.org/jira/browse/HDFS-8643
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8643-00.patch, HDFS-8643-01.patch
>
>
> The idea of this jira to enhance {{SnapshottableDirectoryStatus}} by adding
> {{snapshotNames}} attribute into it, presently it has the {{snapshotNumber}}.
> IMHO this would help the users to get the list of snapshot names created.
> Also, the snapshot names can be used while renaming or deleting the snapshots.
> {code}
> org.apache.hadoop.hdfs.protocol.SnapshottableDirectoryStatus.java
> /**
>* @return Snapshot names for the directory.
>*/
> public List getSnapshotNames() {
> return snapshotNames;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658811#comment-14658811
]
Hadoop QA commented on HDFS-8854:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 56s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 42 new or modified test files. |
| {color:green}+1{color} | javac | 8m 1s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 11s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 40s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 1m 50s | The patch has 5 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 45s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 6m 27s | The patch appears to introduce 8
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 22m 34s | Tests passed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 129m 15s | Tests failed in hadoop-hdfs. |
| {color:red}-1{color} | hdfs tests | 0m 12s | Tests failed in
hadoop-hdfs-client. |
| | | 199m 48s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| FindBugs | module:hadoop-hdfs-client |
| Failed unit tests |
hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots |
| | hadoop.fs.TestUrlStreamHandler |
| | hadoop.hdfs.server.namenode.TestFileLimit |
| | hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot |
| | hadoop.hdfs.server.namenode.TestEditLogAutoroll |
| | hadoop.TestRefreshCallQueue |
| | hadoop.hdfs.protocolPB.TestPBHelper |
| | hadoop.cli.TestCryptoAdminCLI |
| | hadoop.fs.viewfs.TestViewFsWithAcls |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.fs.contract.hdfs.TestHDFSContractDelete |
| | hadoop.fs.TestFcHdfsSetUMask |
| | hadoop.fs.TestUnbuffer |
| | hadoop.hdfs.server.namenode.TestClusterId |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
| | hadoop.hdfs.server.namenode.TestFSDirectory |
| | hadoop.fs.contract.hdfs.TestHDFSContractOpen |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.fs.contract.hdfs.TestHDFSContractMkdir |
| | hadoop.fs.contract.hdfs.TestHDFSContractAppend |
| | hadoop.hdfs.server.namenode.TestSecondaryWebUi |
| | hadoop.hdfs.server.namenode.TestHDFSConcat |
| | hadoop.hdfs.server.namenode.TestAddBlockRetry |
| | hadoop.fs.TestSymlinkHdfsFileSystem |
| | hadoop.fs.viewfs.TestViewFsDefaultValue |
| | hadoop.fs.TestSymlinkHdfsFileContext |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestFSInputChecker |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.cli.TestAclCLI |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional |
| | hadoop.fs.contract.hdfs.TestHDFSContractSeek |
| | hadoop.hdfs.server.namenode.TestFileContextXAttr |
| | hadoop.hdfs.server.namenode.TestAclConfigFlag |
| | hadoop.hdfs.server.namenode.TestFSImageWithXAttr |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot |
| | hadoop.hdfs.server.namenode.TestFSImageWithAcl |
| | hadoop.tracing.TestTracing |
| | hadoop.hdfs.server.namenode.TestListCorruptFileBlocks |
| | hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot |
| | hadoop.hdfs.server.namenode.TestGenericJournalConf |
| | hadoop.fs.viewfs.TestViewFsWithXAttrs |
| | hadoop.cli.TestErasureCodingCLI |
| | hadoop.hdfs.server.namenode.TestEditLogJournalFailures |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.TestDatanodeReport |
| | hadoop.tools.TestJMXGet |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop.fs.contract.hdfs.TestHDFSContractCreate |
| | hadoop.hdfs.server.namenode.TestCreateEditsLog |
| | hadoop.hdfs.server.namenode.TestFSNamesystemMBean |
| | hadoop.hdfs.TestWriteRead |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.fs.TestEnhancedByteBufferAccess |
| | hadoop.fs.TestFcHdfsPerm
[jira] [Updated] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-8856:
Attachment: HDFS-8856.03.patch
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658871#comment-14658871
]
Arpit Agarwal commented on HDFS-8856:
-
I don't see a good reason to return a modifiable reference to the internal list.
bq. One nit: you can use a timeout rule that applies to all cases instead of
adding them individually.
Done in v3 patch.
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[
https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658872#comment-14658872
]
Hadoop QA commented on HDFS-8838:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 22m 25s | Pre-patch HDFS-7285 has 5
extant Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 10m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 44s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 17s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 40s | The applied patch generated 3
new checkstyle issues (total was 172, now 168). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 56s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 40s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 12s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 48s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 78m 20s | Tests failed in hadoop-hdfs. |
| | | 136m 46s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.TestRead |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.namenode.TestSecondaryWebUi |
| | hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional |
| | hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd |
| | hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
| | hadoop.hdfs.server.datanode.TestRefreshNamenodes |
| | hadoop.hdfs.TestHdfsAdmin |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
| | hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler |
| | hadoop.hdfs.TestClientReportBadBlock |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.namenode.TestAuditLogger |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy |
| | hadoop.hdfs.server.blockmanagement.TestDatanodeManager |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.TestWriteStripedFileWithFailure |
| |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement |
| | hadoop.hdfs.server.namenode.TestNameNodeRpcServer |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade |
| | hadoop.cli.TestErasureCodingCLI |
| | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot |
| | hadoop.hdfs.server.namenode.ha.TestHAFsck |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
| | hadoop.hdfs.server.namenode.TestHDFSConcat |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.TestRefreshCallQueue |
| | hadoop.hdfs.TestListFilesInDFS |
| | hadoop.hdfs.server.namenode.TestAddBlock |
| | hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold |
| | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
| | hadoop.hdfs.TestMiniDFSCluster |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.security.TestPermissionSymlinks |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica |
| | hadoop.hdfs.TestFileConcurrentReader |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.server.datanode.TestDataNodeExit |
| | hadoop.hdfs.server.blockmanagement.TestSequentialBloc
[jira] [Commented] (HDFS-8696) Reduce the variances of latency of WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658875#comment-14658875
]
Jun Yin commented on HDFS-8696:
---
On unpatched try
$ curl -s -L
"http://NN:50070/webhdfs/v1/tmp/catalog_sales_38_50.dat?op=OPEN&user.name=release&length=1";,
will get data which length=1
While on patched cluster it will get whole data just as same as execution
$ curl -s -L
"http://NN:50070/webhdfs/v1/tmp/catalog_sales_38_50.dat?op=OPEN&user.name=release";
please try if you can also reproduce, maybe i missed something.
> Reduce the variances of latency of WebHDFS
> --
>
> Key: HDFS-8696
> URL: https://issues.apache.org/jira/browse/HDFS-8696
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-8696.1.patch, HDFS-8696.2.patch, HDFS-8696.3.patch
>
>
> There is an issue that appears related to the webhdfs server. When making two
> concurrent requests, the DN will sometimes pause for extended periods (I've
> seen 1-300 seconds), killing performance and dropping connections.
> To reproduce:
> 1. set up a HDFS cluster
> 2. Upload a large file (I was using 10GB). Perform 1-byte reads, writing
> the time out to /tmp/times.txt
> {noformat}
> i=1
> while (true); do
> echo $i
> let i++
> /usr/bin/time -f %e -o /tmp/times.txt -a curl -s -L -o /dev/null
> "http://:50070/webhdfs/v1/tmp/bigfile?op=OPEN&user.name=root&length=1";
> done
> {noformat}
> 3. Watch for 1-byte requests that take more than one second:
> tail -F /tmp/times.txt | grep -E "^[^0]"
> 4. After it has had a chance to warm up, start doing large transfers from
> another shell:
> {noformat}
> i=1
> while (true); do
> echo $i
> let i++
> (/usr/bin/time -f %e curl -s -L -o /dev/null
> "http://:50070/webhdfs/v1/tmp/bigfile?op=OPEN&user.name=root");
> done
> {noformat}
> It's easy to find after a minute or two that small reads will sometimes
> pause for 1-300 seconds. In some extreme cases, it appears that the
> transfers timeout and the DN drops the connection.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658902#comment-14658902
]
Andrew Wang commented on HDFS-8833:
---
I think EC schema differs from storage policies since the useful configuration
space is much smaller. This is based on how other storage systems with EC
support only a single (or very few) schemas. QFS I believe hardcodes 6,3 with a
64KB cell size. Similarly, f4 does just 10,4. Even if we were to provide all
the EC schemas implemented by these other storage systems, I bet we'd be well
within 64 (and probably even well within 16). I think 64 would give us plenty
of room even when adding new codecs like Hitchhiker or LRC.
For directory rename, the idea is we set the EC policy on the directory? This
feels a little weird (and different from storage policies) since it means
renaming a subdir implicitly calls createECZone or whatever on the subdir, i.e.:
{noformat}
/ec1 <-- ec policy 1
/ec2 <-- ec policy 2
rename /ec2/subdir to /ec1/subdir
create new file /ec1/subdir/foo
/ec1/subdir/foo will get policy 2, not policy 1
{noformat}
We could maybe solve this by having the directory-rename specify "schema of
contained files" but not specify the schema of newly created files...but then
we need to attach schema info on new files since they differ from the
directory's info. More complicated for sure, I haven't thought through all the
cases.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-8808) dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby
[
https://issues.apache.org/jira/browse/HDFS-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang reassigned HDFS-8808:
---
Assignee: Zhe Zhang
> dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby
>
>
> Key: HDFS-8808
> URL: https://issues.apache.org/jira/browse/HDFS-8808
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Gautam Gopalakrishnan
>Assignee: Zhe Zhang
>
> The parameter {{dfs.image.transfer.bandwidthPerSec}} can be used to limit the
> speed with which the fsimage is copied between the namenodes during regular
> use. However, as a side effect, this also limits transfers when the
> {{-bootstrapStandby}} option is used. This option is often used during
> upgrades and could potentially slow down the entire workflow. The request
> here is to ensure {{-bootstrapStandby}} is unaffected by this bandwidth
> setting
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8808) dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby
[
https://issues.apache.org/jira/browse/HDFS-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658936#comment-14658936
]
Zhe Zhang commented on HDFS-8808:
-
Thanks [~ggop] for reporting the problem.
[~ajithshetty]: I think your idea is a valid workaround. However I think we
still need a proper fix to address the issue. Changing system configuration
only for a specific operation incurs more burden to admins and is prone to
human errors.
I'll take a crack on the task.
> dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby
>
>
> Key: HDFS-8808
> URL: https://issues.apache.org/jira/browse/HDFS-8808
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Gautam Gopalakrishnan
>Assignee: Zhe Zhang
>
> The parameter {{dfs.image.transfer.bandwidthPerSec}} can be used to limit the
> speed with which the fsimage is copied between the namenodes during regular
> use. However, as a side effect, this also limits transfers when the
> {{-bootstrapStandby}} option is used. This option is often used during
> upgrades and could potentially slow down the entire workflow. The request
> here is to ensure {{-bootstrapStandby}} is unaffected by this bandwidth
> setting
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8775) SASL support for data transfer protocol in libhdfspp
[ https://issues.apache.org/jira/browse/HDFS-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658938#comment-14658938 ] Haohui Mai commented on HDFS-8775: -- bq. I think it would make things more maintainable to pick one and stick with it I followed the Google's style here. References are used for const strings while pointers to string indicate the strings are mutable. > SASL support for data transfer protocol in libhdfspp > > > Key: HDFS-8775 > URL: https://issues.apache.org/jira/browse/HDFS-8775 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-8775.000.patch > > > This jira proposes to implement basic SASL support for the data transfer > protocol which allows libhdfspp to talk to secure clusters. > Support for encryption is deferred to subsequent jiras. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658940#comment-14658940
]
Haohui Mai commented on HDFS-8856:
--
+1 pending jenkins.
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7597) DNs should not open new NN connections when webhdfs clients seek
[
https://issues.apache.org/jira/browse/HDFS-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658962#comment-14658962
]
Hadoop QA commented on HDFS-7597:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 52s | Findbugs (version 3.0.0)
appears to be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 59s | The applied patch generated 1
new checkstyle issues (total was 11, now 12). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 22s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 4s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 162m 42s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 26s | Tests passed in
hadoop-hdfs-client. |
| | | 209m 11s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.server.namenode.ha.TestStandbyIsHot |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748885/HDFS-7597.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4ab49a4 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11907/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11907/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11907/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11907/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11907/console |
This message was automatically generated.
> DNs should not open new NN connections when webhdfs clients seek
>
>
> Key: HDFS-7597
> URL: https://issues.apache.org/jira/browse/HDFS-7597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Labels: BB2015-05-TBR
> Attachments: HDFS-7597.patch, HDFS-7597.patch, HDFS-7597.patch
>
>
> Webhdfs seeks involve closing the current connection, and reissuing a new
> open request with the new offset. The RPC layer caches connections so the DN
> keeps a lingering connection open to the NN. Connection caching is in part
> based on UGI. Although the client used the same token for the new offset
> request, the UGI is different which forces the DN to open another unnecessary
> connection to the NN.
> A job that performs many seeks will easily crash the NN due to fd exhaustion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658979#comment-14658979
]
Hadoop QA commented on HDFS-8856:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 25s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 45s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 25s | The applied patch generated 2
new checkstyle issues (total was 296, now 293). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 12 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 21s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 33s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 4s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 47s | Tests passed in hadoop-hdfs.
|
| | | 207m 8s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748887/HDFS-8856.02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4ab49a4 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11908/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11908/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11908/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11908/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11908/console |
This message was automatically generated.
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658978#comment-14658978
]
Jing Zhao commented on HDFS-8833:
-
bq. QFS I believe hardcodes 6,3 with a 64KB cell size. Similarly, f4 does just
10,4. Even if we were to provide all the EC schemas implemented by these other
storage systems, I bet we'd be well within 64 (and probably even well within
16).
To me we should either hard code these numbers and not allow clients to add new
policies, or we avoid some unnecessary limitation in the very beginning. The
total number of policies can easily exceed 64 if the policy includes cell size,
as proposed here and in HDFS-8854.
bq. When renamed, a directory carries over its ECPolicy if it's set (XAttr
non-empty).
bq. For directory rename, the idea is we set the EC policy on the directory?
This feels a little weird (and different from storage policies) since it means
renaming a subdir implicitly calls createECZone or whatever on the subdir
Yes, the nested EC policy will definitely bring extra complexity and make it
hard for admins to understand and manage the cluster. If we cannot figure out a
perfect solution immediately, why not keeping the current zone implementation
first? Considering the main use case of the first stage is just storing cold
data, we will not hit too many issues because of the rename restriction. Then
based on user's feedback we can consider if we want to remove the restriction
or bring in the concept of HDFS Volume. Maybe [~sanjay.radia] can also comment
here.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8486) DN startup may cause severe data loss
[ https://issues.apache.org/jira/browse/HDFS-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659030#comment-14659030 ] Arpit Agarwal commented on HDFS-8486: - Thanks [~xyao], will hold off committing for a couple of days in case there are additional comments. > DN startup may cause severe data loss > - > > Key: HDFS-8486 > URL: https://issues.apache.org/jira/browse/HDFS-8486 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.1, 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.7.1 > > Attachments: HDFS-8486-branch-2.6.02.patch, > HDFS-8486-branch-2.6.patch, HDFS-8486.patch, HDFS-8486.patch > > > A race condition between block pool initialization and the directory scanner > may cause a mass deletion of blocks in multiple storages. > If block pool initialization finds a block on disk that is already in the > replica map, it deletes one of the blocks based on size, GS, etc. > Unfortunately it _always_ deletes one of the blocks even if identical, thus > the replica map _must_ be empty when the pool is initialized. > The directory scanner starts at a random time within its periodic interval > (default 6h). If the scanner starts very early it races to populate the > replica map, causing the block pool init to erroneously delete blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659049#comment-14659049
]
Zhe Zhang commented on HDFS-8499:
-
Thanks!
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0
>
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for striped and contiguous UC blocks. This JIRA aims to
> merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659059#comment-14659059
]
Andrew Wang commented on HDFS-8833:
---
Why does cell size blow up the configuration space? AFAIK values used by other
systems are 64KB and 1MB. So it potentially doubles whatever we had before, but
I think that still fits within 64 easily, e.g.
* Schema: RS
* Parameters: (3,2) (6,3) (10,4)
* Cell size: 64KB 1MB
This is just 6. Even if we add LRC and Hitchhiker later on and that comes to
18. Seems future-proof enough even with just a few bits.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659078#comment-14659078
]
Jing Zhao commented on HDFS-8833:
-
Then how can you make sure users only set 64KB/1MB ? Even during EC development
the cell size was set to 256 KB for a while.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659087#comment-14659087
]
Andrew Wang commented on HDFS-8833:
---
Weren't we planning to hardcode the available schemas for phase 1? Users would
just choose from a table like how storage policies work right now. In phase 2
with pluggable codecs we can always extend with an xattr for additional space
if needed (the hybrid scheme we've been talking about).
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659112#comment-14659112
]
Jing Zhao commented on HDFS-8833:
-
Then why not just hardcode one schema for phase 1 (which is the current
implementation), and leave all the extension to phase 2? As you mentioned most
existing systems hard code their schema and cell size. Why do we want to occupy
multiple bits in INodeFile header and later still use xattr as extension?
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8860) Remove Replica hardlink / unlink code
Lei (Eddy) Xu created HDFS-8860:
---
Summary: Remove Replica hardlink / unlink code
Key: HDFS-8860
URL: https://issues.apache.org/jira/browse/HDFS-8860
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
{{ReplicaInfo#unlinkBlock()}} is effectively disabled by the following code,
because {{isUnlinked()}} always returns true.
{code}
if (isUnlinked()) {
return false;
}
{code}
Several test cases, e.g., {{TestFileAppend#testCopyOnWrite}} and
{{TestDatanodeRestart#testRecoverReplicas}} are testing against the unlink
Lets remove the relevant code to eliminate the confusions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
Xiaobing Zhou created HDFS-8861: --- Summary: Remove unnecessary log from method FSNamesystem.getCorruptFiles Key: HDFS-8861 URL: https://issues.apache.org/jira/browse/HDFS-8861 Project: Hadoop HDFS Issue Type: Improvement Components: HDFS Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou The log in FSNamesystem.getCorruptFiles will print out whole stack trace. In those cases where SuperuserPrivilege check and Operation check are not satisfied in frequent calls of listCorruptFileBlocks, the stack traces will make logs quite verbose, hard to understand and analyze it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
[ https://issues.apache.org/jira/browse/HDFS-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-8861: Attachment: HDFS-8861.1.patch Made a patch V1, simply removing the log print. > Remove unnecessary log from method FSNamesystem.getCorruptFiles > --- > > Key: HDFS-8861 > URL: https://issues.apache.org/jira/browse/HDFS-8861 > Project: Hadoop HDFS > Issue Type: Improvement > Components: HDFS >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-8861.1.patch > > > The log in FSNamesystem.getCorruptFiles will print out whole stack trace. In > those cases where SuperuserPrivilege check and Operation check are not > satisfied in frequent calls of listCorruptFileBlocks, the stack traces will > make logs quite verbose, hard to understand and analyze it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659128#comment-14659128
]
Andrew Wang commented on HDFS-8833:
---
Good question, I think it basically comes down to our deployment scenarios
being more broad than Quantcast or Facebook. You want a # of racks equal to the
stripe width for fault tolerance. FB and Quantcast are big enough that they run
14 rack or 9 rack clusters, but not all of our customers are at that same
scale. So there isn't a one-size-fits-all schema that works for all HDFS users;
the big ones will use (10,4) or (6,3) like FB and Quantcast, but the smaller
ones will want (3,2).
I've also seen customers starting with small clusters and growing them by
adding racks over time. This is also somewhat unique to HDFS compared to QFS
and f4, and a reason why it'd be nice to support a few different policies even
within the same cluster.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8643) Add snapshot names list to SnapshottableDirectoryStatus
[
https://issues.apache.org/jira/browse/HDFS-8643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659131#comment-14659131
]
Hadoop QA commented on HDFS-8643:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 51s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 29s | The applied patch generated 1
new checkstyle issues (total was 15, now 15). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 24s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 1s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 161m 55s | Tests passed in hadoop-hdfs.
|
| {color:green}+1{color} | hdfs tests | 0m 28s | Tests passed in
hadoop-hdfs-client. |
| | | 210m 54s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748900/HDFS-8643-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4ab49a4 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11910/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11910/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11910/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11910/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11910/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11910/console |
This message was automatically generated.
> Add snapshot names list to SnapshottableDirectoryStatus
> ---
>
> Key: HDFS-8643
> URL: https://issues.apache.org/jira/browse/HDFS-8643
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8643-00.patch, HDFS-8643-01.patch
>
>
> The idea of this jira to enhance {{SnapshottableDirectoryStatus}} by adding
> {{snapshotNames}} attribute into it, presently it has the {{snapshotNumber}}.
> IMHO this would help the users to get the list of snapshot names created.
> Also, the snapshot names can be used while renaming or deleting the snapshots.
> {code}
> org.apache.hadoop.hdfs.protocol.SnapshottableDirectoryStatus.java
> /**
>* @return Snapshot names for the directory.
>*/
> public List getSnapshotNames() {
> return snapshotNames;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
[ https://issues.apache.org/jira/browse/HDFS-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-8861: Status: Patch Available (was: Open) > Remove unnecessary log from method FSNamesystem.getCorruptFiles > --- > > Key: HDFS-8861 > URL: https://issues.apache.org/jira/browse/HDFS-8861 > Project: Hadoop HDFS > Issue Type: Improvement > Components: HDFS >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-8861.1.patch > > > The log in FSNamesystem.getCorruptFiles will print out whole stack trace. In > those cases where SuperuserPrivilege check and Operation check are not > satisfied in frequent calls of listCorruptFileBlocks, the stack traces will > make logs quite verbose, hard to understand and analyze it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659150#comment-14659150
]
Jing Zhao commented on HDFS-8833:
-
I understand this, but still thanks for the explanation, Andrew. I'm not
opposing to support multiple EC schemas. My point is we should only use xattr
for representing ec policy for file/dir, instead of occupying several bits in
INodeFile first then coming to xattr later.
Also, instead of doing this extension at this stage, I think it's better to
support one schema only first, and spend more time on testing and stabilizing
the current implementation, especially considering we still have some issues
with the write code path.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8827) Erasure Coding: When namenode processes over replicated striped block, NPE will be occur in ReplicationMonitor
[
https://issues.apache.org/jira/browse/HDFS-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8827:
Attachment: HDFS-8827.2.patch
Thanks [~tfukudom] for working on this! I agree with Walter that for the unit
test, the NPE is more like caused by the testing code. The .2 patch writes the
real data instead of using {{createStripedFile}}, but it exposes some other
bugs in the current writing code path so I have to change the block size from 1
cell to 4 cells. In the meanwhile the test passed with the change. So the NPE
you saw in the system test may be caused by some other issue. I will dig
further into that.
> Erasure Coding: When namenode processes over replicated striped block, NPE
> will be occur in ReplicationMonitor
> --
>
> Key: HDFS-8827
> URL: https://issues.apache.org/jira/browse/HDFS-8827
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takuya Fukudome
>Assignee: Takuya Fukudome
> Attachments: HDFS-8827.1.patch, HDFS-8827.2.patch,
> processing-over-replica-npe.log
>
>
> In our test cluster, when namenode processed over replicated striped blocks,
> null pointer exception(NPE) occurred. This happened under below situation: 1)
> some datanodes shutdown. 2) namenode recovers block group which lost internal
> blocks. 3) restart the stopped datanodes. 4) namenode processes over
> replicated striped blocks. 5) NPE occurs
> I think BlockPlacementPolicyDefault#chooseReplicaToDelete will return null in
> this situation which causes this NPE problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659173#comment-14659173
]
Andrew Wang commented on HDFS-8833:
---
bq. My point is we should only use xattr for representing ec policy for
file/dir, instead of occupying several bits in INodeFile first then coming to
xattr later.
I guess this is the core of the issue, since I don't understand your objection
to using file header bits. Are you worried that we will want to use these bits
for another purpose? Or that the code to later use an xattr will be complex?
Would like to discuss this since it seems like the focal point and influences
our implementation choices.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659189#comment-14659189
]
Zhe Zhang commented on HDFS-8854:
-
Some high level comments:
# Should we split out the patch in common?
# Per discussion under HDFS-8129, in hdfs package let's use
{{ErasureCodingPolicy}} instead off {{ECPolicy}}
# Can we split out {{ECPolicyLoader}} as follow-on? [~drankye] This class is
actually not used by other classes except for a test. Did I miss anything?
The rest looks like clean refactor. Will review again after addressing the
above. Thanks!
> Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
> --
>
> Key: HDFS-8854
> URL: https://issues.apache.org/jira/browse/HDFS-8854
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8854-HDFS-7285.00.patch, HDFS-8854.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8772) fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659198#comment-14659198
]
Andrew Wang commented on HDFS-8772:
---
Sure, the point of test timeouts is more so the build doesn't hang for hours,
not to be particularly stringent about them. Anyways thanks for the work here
all, I'll commit shortly.
> fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
>
>
> Key: HDFS-8772
> URL: https://issues.apache.org/jira/browse/HDFS-8772
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8772.01.patch, HDFS-8772.02.patch,
> HDFS-8772.03.patch, HDFS-8772.04.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/11596/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11598/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11599/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11600/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11606/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11608/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11612/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11618/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11650/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11655/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11659/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11663/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11664/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11667/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11669/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11676/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11677/testReport/
> {noformat}
> java.lang.AssertionError: expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot.testDatanodeRestarts(TestStandbyIsHot.java:188)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8772) Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-8772:
--
Summary: Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
(was: fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails )
> Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
>
>
> Key: HDFS-8772
> URL: https://issues.apache.org/jira/browse/HDFS-8772
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8772.01.patch, HDFS-8772.02.patch,
> HDFS-8772.03.patch, HDFS-8772.04.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/11596/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11598/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11599/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11600/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11606/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11608/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11612/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11618/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11650/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11655/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11659/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11663/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11664/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11667/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11669/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11676/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11677/testReport/
> {noformat}
> java.lang.AssertionError: expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot.testDatanodeRestarts(TestStandbyIsHot.java:188)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8851) datanode fails to start due to a bad disk
[ https://issues.apache.org/jira/browse/HDFS-8851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659223#comment-14659223 ] Andrew Wang commented on HDFS-8851: --- Is this a dupe of HDFS-8636? I have a half-done patch for that I haven't gotten around to, but you can feel free to take it over if you want. > datanode fails to start due to a bad disk > - > > Key: HDFS-8851 > URL: https://issues.apache.org/jira/browse/HDFS-8851 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.1 >Reporter: Wang Hao > > Data node can not start due to a bad disk. I found a similar issue HDFS-6245 > is reported, but our situation is different. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8772) Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659229#comment-14659229
]
Andrew Wang commented on HDFS-8772:
---
Ran into some issues backporting to branch-2, [~walter.k.su] mind preparing
another patch for branch-2? We conflict a bit with HDFS-6440 in MiniDFSCluster,
hoping we can fix by including some of those refactors.
> Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails
>
>
> Key: HDFS-8772
> URL: https://issues.apache.org/jira/browse/HDFS-8772
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8772.01.patch, HDFS-8772.02.patch,
> HDFS-8772.03.patch, HDFS-8772.04.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/11596/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11598/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11599/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11600/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11606/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11608/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11612/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11618/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11650/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11655/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11659/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11663/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11664/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11667/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11669/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11676/testReport/
> https://builds.apache.org/job/PreCommit-HDFS-Build/11677/testReport/
> {noformat}
> java.lang.AssertionError: expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot.testDatanodeRestarts(TestStandbyIsHot.java:188)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8845) DiskChecker should not traverse entire tree
[ https://issues.apache.org/jira/browse/HDFS-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659244#comment-14659244 ] Andrew Wang commented on HDFS-8845: --- It'd be a little better for coverage if we tried looking at the file path that had the error, since it's possible there's an error on some subdir. Agree though that traversing the entire tree doesn't have much point though. > DiskChecker should not traverse entire tree > --- > > Key: HDFS-8845 > URL: https://issues.apache.org/jira/browse/HDFS-8845 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: HDFS-8845.patch > > > DiskChecker should not traverse entire tree because it's causing heavy disk > load on checkDiskError() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659261#comment-14659261
]
Yi Liu commented on HDFS-8856:
--
{quote}
Looks risky, the lock ensures consistent state for each INodeFile.
{quote}
Arpit, sorry, I mean to put following in the {{try .. finally}} closure,
ideally it will not throw exception in real case, but from the implementation,
it contains.
{code}
numUCBlocks = leaseManager.getNumUnderConstructionBlocks();
{code}
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659261#comment-14659261
]
Yi Liu edited comment on HDFS-8856 at 8/6/15 12:36 AM:
---
{quote}
Looks risky, the lock ensures consistent state for each INodeFile.
{quote}
Arpit, sorry, I mean to put following in the {{try .. finally}} closure,
ideally it will not throw exception in real case, but from the implementation,
it contains and could cause dead lock.
{code}
numUCBlocks = leaseManager.getNumUnderConstructionBlocks();
{code}
was (Author: hitliuyi):
{quote}
Looks risky, the lock ensures consistent state for each INodeFile.
{quote}
Arpit, sorry, I mean to put following in the {{try .. finally}} closure,
ideally it will not throw exception in real case, but from the implementation,
it contains.
{code}
numUCBlocks = leaseManager.getNumUnderConstructionBlocks();
{code}
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8499:
Attachment: revert-HDFS-8499-HDFS-8623.patch
Upload a patch for reverting both HDFS-8499 and HDFS-8623. I hit some conflicts
when reverting the changes, so let's run Jenkins to verify.
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0
>
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch,
> revert-HDFS-8499-HDFS-8623.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for striped and contiguous UC blocks. This JIRA aims to
> merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8499:
Status: Patch Available (was: Reopened)
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0
>
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch,
> revert-HDFS-8499-HDFS-8623.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for striped and contiguous UC blocks. This JIRA aims to
> merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao reopened HDFS-8499:
-
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0
>
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch,
> revert-HDFS-8499-HDFS-8623.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for striped and contiguous UC blocks. This JIRA aims to
> merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8856) Make LeaseManager#countPath O(1)
[
https://issues.apache.org/jira/browse/HDFS-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659282#comment-14659282
]
Hadoop QA commented on HDFS-8856:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 53s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 13s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 54s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 28s | The applied patch generated 2
new checkstyle issues (total was 296, now 293). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 21s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 162m 14s | Tests failed in hadoop-hdfs. |
| | | 208m 48s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestReplaceDatanodeOnFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748920/HDFS-8856.03.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ba2313d |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11911/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11911/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11911/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11911/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11911/console |
This message was automatically generated.
> Make LeaseManager#countPath O(1)
>
>
> Key: HDFS-8856
> URL: https://issues.apache.org/jira/browse/HDFS-8856
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8856.01.patch, HDFS-8856.02.patch,
> HDFS-8856.03.patch
>
>
> {{LeaseManager#countPath}} loops over all existing lease holders to compute
> the pending lease count. We can just track the pending leased files so it
> runs in constant time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8792) Improve BlockManager#postponedMisreplicatedBlocks and BlockManager#excessReplicateMap
[
https://issues.apache.org/jira/browse/HDFS-8792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659302#comment-14659302
]
Yi Liu commented on HDFS-8792:
--
Thanks [~cmccabe] for the review.
{quote}
This JavaDoc comment is stale. It seems like it should be "the current
modification epoch" or similar.
{quote}
Will update it in the new patch right now.
{quote}
Can you put this in a separate JIRA? It's less obvious to me that this is a
win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when
elements are removed.
{quote}
Sure, I will put it in a separate JIRA.
That's right HashMap don't ever shrink when elements are removed, but TreeMap
entry needs to store more (memory) references (left, right, parent) than
HashMap entry (only one reference next), even when there is element removing,
the empty HashMap entry is just a {{null}} reference (4 bytes), so they are
nearly at this point. On the other hand, the key of {{excessReplicateMap}} is
datanode uuid, so the entries number is almost fixed, so HashMap memory is good
than TreeMap memory in this case. I think the most important is the
search/insert/remove performance, HashMap is absolutely better than TreeMap.
Because we don't need to sort, we should use HashMap instead of TreeMap.
> Improve BlockManager#postponedMisreplicatedBlocks and
> BlockManager#excessReplicateMap
> -
>
> Key: HDFS-8792
> URL: https://issues.apache.org/jira/browse/HDFS-8792
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8792.001.patch, HDFS-8792.002.patch
>
>
> {{LightWeightHashSet}} requires fewer memory than java hashset.
> Furthermore, for {{excessReplicateMap}}, we can use {{HashMap}} instead of
> {{TreeMap}} instead, since no need to sort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8792) Improve BlockManager#postponedMisreplicatedBlocks and BlockManager#excessReplicateMap
[
https://issues.apache.org/jira/browse/HDFS-8792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-8792:
-
Attachment: HDFS-8792.003.patch
Update the new patch.
> Improve BlockManager#postponedMisreplicatedBlocks and
> BlockManager#excessReplicateMap
> -
>
> Key: HDFS-8792
> URL: https://issues.apache.org/jira/browse/HDFS-8792
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8792.001.patch, HDFS-8792.002.patch,
> HDFS-8792.003.patch
>
>
> {{LightWeightHashSet}} requires fewer memory than java hashset.
> Furthermore, for {{excessReplicateMap}}, we can use {{HashMap}} instead of
> {{TreeMap}} instead, since no need to sort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8792) Improve BlockManager#postponedMisreplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-8792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-8792:
-
Summary: Improve BlockManager#postponedMisreplicatedBlocks (was: Improve
BlockManager#postponedMisreplicatedBlocks and BlockManager#excessReplicateMap)
> Improve BlockManager#postponedMisreplicatedBlocks
> -
>
> Key: HDFS-8792
> URL: https://issues.apache.org/jira/browse/HDFS-8792
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8792.001.patch, HDFS-8792.002.patch,
> HDFS-8792.003.patch
>
>
> {{LightWeightHashSet}} requires fewer memory than java hashset.
> Furthermore, for {{excessReplicateMap}}, we can use {{HashMap}} instead of
> {{TreeMap}} instead, since no need to sort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8827) Erasure Coding: When namenode processes over replicated striped block, NPE will be occur in ReplicationMonitor
[ https://issues.apache.org/jira/browse/HDFS-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659310#comment-14659310 ] Takuya Fukudome commented on HDFS-8827: --- Thanks for your comments, [~walter.k.su], [~jingzhao]! I understood that my test code caused this NPE. In our test cluster, Namenode seems to have processed 2 redundant blocks and 1 missing block at the same time. If I can get more specific logs, I will upload it. Thank you. > Erasure Coding: When namenode processes over replicated striped block, NPE > will be occur in ReplicationMonitor > -- > > Key: HDFS-8827 > URL: https://issues.apache.org/jira/browse/HDFS-8827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Takuya Fukudome >Assignee: Takuya Fukudome > Attachments: HDFS-8827.1.patch, HDFS-8827.2.patch, > processing-over-replica-npe.log > > > In our test cluster, when namenode processed over replicated striped blocks, > null pointer exception(NPE) occurred. This happened under below situation: 1) > some datanodes shutdown. 2) namenode recovers block group which lost internal > blocks. 3) restart the stopped datanodes. 4) namenode processes over > replicated striped blocks. 5) NPE occurs > I think BlockPlacementPolicyDefault#chooseReplicaToDelete will return null in > this situation which causes this NPE problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8862) Improve BlockManager#excessReplicateMap
Yi Liu created HDFS-8862:
Summary: Improve BlockManager#excessReplicateMap
Key: HDFS-8862
URL: https://issues.apache.org/jira/browse/HDFS-8862
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Per [~cmccabe]'s comments in HDFS-8792, this JIRA is to discuss improving
{{BlockManager#excessReplicateMap}}.
That's right HashMap don't ever shrink when elements are removed, but TreeMap
entry needs to store more (memory) references (left, right, parent) than
HashMap entry (only one reference next), even when there is element removing
and cause some entry empty, the empty HashMap entry is just a {{null}}
reference (4 bytes), so they are close at this point. On the other hand, the
key of {{excessReplicateMap}} is datanode uuid, so the entries number is almost
fixed, so HashMap memory is good than TreeMap memory in this case. I think
the most important is the search/insert/remove performance, HashMap is
absolutely better than TreeMap. Because we don't need to sort, we should use
HashMap instead of TreeMap
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-8792) Improve BlockManager#postponedMisreplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-8792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659302#comment-14659302
]
Yi Liu edited comment on HDFS-8792 at 8/6/15 1:32 AM:
--
Thanks [~cmccabe] for the review.
{quote}
This JavaDoc comment is stale. It seems like it should be "the current
modification epoch" or similar.
{quote}
Will update it in the new patch right now.
{quote}
Can you put this in a separate JIRA? It's less obvious to me that this is a
win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when
elements are removed.
{quote}
Sure, I will put it in a separate JIRA.
That's right HashMap don't ever shrink when elements are removed, but TreeMap
entry needs to store more (memory) references (left, right, parent) than
HashMap entry (only one reference next), even when there is element removing
and cause some entry empty, the empty HashMap entry is just a {{null}}
reference (4 bytes), so they are close at this point. On the other hand, the
key of {{excessReplicateMap}} is datanode uuid, so the entries number is almost
fixed, so HashMap memory is good than TreeMap memory in this case. I think
the most important is the search/insert/remove performance, HashMap is
absolutely better than TreeMap. Because we don't need to sort, we should use
HashMap instead of TreeMap.
was (Author: hitliuyi):
Thanks [~cmccabe] for the review.
{quote}
This JavaDoc comment is stale. It seems like it should be "the current
modification epoch" or similar.
{quote}
Will update it in the new patch right now.
{quote}
Can you put this in a separate JIRA? It's less obvious to me that this is a
win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when
elements are removed.
{quote}
Sure, I will put it in a separate JIRA.
That's right HashMap don't ever shrink when elements are removed, but TreeMap
entry needs to store more (memory) references (left, right, parent) than
HashMap entry (only one reference next), even when there is element removing,
the empty HashMap entry is just a {{null}} reference (4 bytes), so they are
nearly at this point. On the other hand, the key of {{excessReplicateMap}} is
datanode uuid, so the entries number is almost fixed, so HashMap memory is good
than TreeMap memory in this case. I think the most important is the
search/insert/remove performance, HashMap is absolutely better than TreeMap.
Because we don't need to sort, we should use HashMap instead of TreeMap.
> Improve BlockManager#postponedMisreplicatedBlocks
> -
>
> Key: HDFS-8792
> URL: https://issues.apache.org/jira/browse/HDFS-8792
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8792.001.patch, HDFS-8792.002.patch,
> HDFS-8792.003.patch
>
>
> {{LightWeightHashSet}} requires fewer memory than java hashset.
> Furthermore, for {{excessReplicateMap}}, we can use {{HashMap}} instead of
> {{TreeMap}} instead, since no need to sort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8792) Improve BlockManager#postponedMisreplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-8792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659321#comment-14659321
]
Yi Liu commented on HDFS-8792:
--
I move {{BlockManager#excessReplicateMap}} to HDFS-8862.
> Improve BlockManager#postponedMisreplicatedBlocks
> -
>
> Key: HDFS-8792
> URL: https://issues.apache.org/jira/browse/HDFS-8792
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8792.001.patch, HDFS-8792.002.patch,
> HDFS-8792.003.patch
>
>
> {{LightWeightHashSet}} requires fewer memory than java hashset.
> Furthermore, for {{excessReplicateMap}}, we can use {{HashMap}} instead of
> {{TreeMap}} instead, since no need to sort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8862) Improve BlockManager#excessReplicateMap
[
https://issues.apache.org/jira/browse/HDFS-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-8862:
-
Attachment: HDFS-8862.001.patch
> Improve BlockManager#excessReplicateMap
> ---
>
> Key: HDFS-8862
> URL: https://issues.apache.org/jira/browse/HDFS-8862
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8862.001.patch
>
>
> Per [~cmccabe]'s comments in HDFS-8792, this JIRA is to discuss improving
> {{BlockManager#excessReplicateMap}}.
> That's right HashMap don't ever shrink when elements are removed, but
> TreeMap entry needs to store more (memory) references (left, right, parent)
> than HashMap entry (only one reference next), even when there is element
> removing and cause some entry empty, the empty HashMap entry is just a
> {{null}} reference (4 bytes), so they are close at this point. On the other
> hand, the key of {{excessReplicateMap}} is datanode uuid, so the entries
> number is almost fixed, so HashMap memory is good than TreeMap memory in this
> case. I think the most important is the search/insert/remove performance,
> HashMap is absolutely better than TreeMap. Because we don't need to sort,
> we should use HashMap instead of TreeMap
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8862) Improve BlockManager#excessReplicateMap
[
https://issues.apache.org/jira/browse/HDFS-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-8862:
-
Status: Patch Available (was: Open)
> Improve BlockManager#excessReplicateMap
> ---
>
> Key: HDFS-8862
> URL: https://issues.apache.org/jira/browse/HDFS-8862
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8862.001.patch
>
>
> Per [~cmccabe]'s comments in HDFS-8792, this JIRA is to discuss improving
> {{BlockManager#excessReplicateMap}}.
> That's right HashMap don't ever shrink when elements are removed, but
> TreeMap entry needs to store more (memory) references (left, right, parent)
> than HashMap entry (only one reference next), even when there is element
> removing and cause some entry empty, the empty HashMap entry is just a
> {{null}} reference (4 bytes), so they are close at this point. On the other
> hand, the key of {{excessReplicateMap}} is datanode uuid, so the entries
> number is almost fixed, so HashMap memory is good than TreeMap memory in this
> case. I think the most important is the search/insert/remove performance,
> HashMap is absolutely better than TreeMap. Because we don't need to sort,
> we should use HashMap instead of TreeMap
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659325#comment-14659325 ] Walter Su commented on HDFS-8854: - bq. Should we split out the patch in common? Can't run jenkins because of dependence. bq. Per discussion under HDFS-8129, in hdfs package let's use ErasureCodingPolicy instead off ECPolicy I'll do it. bq. Can we split out ECPolicyLoader as follow-on? Since we're in discussion about hard-coded vs customized policy. I'm ok with that. [~drankye], how do you think? > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Walter Su > Attachments: HDFS-8854-HDFS-7285.00.patch, HDFS-8854.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8859) Improve DataNode (ReplicaMap) memory footprint to save about 45%
[
https://issues.apache.org/jira/browse/HDFS-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-8859:
-
Priority: Critical (was: Major)
> Improve DataNode (ReplicaMap) memory footprint to save about 45%
>
>
> Key: HDFS-8859
> URL: https://issues.apache.org/jira/browse/HDFS-8859
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Critical
>
> By using following approach we can save about *45%* memory footprint for each
> block replica in DataNode memory (This JIRA only talks about *ReplicaMap* in
> DataNode), the details are:
> In ReplicaMap,
> {code}
> private final Map> map =
> new HashMap>();
> {code}
> Currently we use a HashMap {{Map}} to store the replicas
> in memory. The key is block id of the block replica which is already
> included in {{ReplicaInfo}}, so this memory can be saved. Also HashMap Entry
> has a object overhead. We can implement a lightweight Set which is similar
> to {{LightWeightGSet}}, but not a fixed size ({{LightWeightGSet}} uses fix
> size for the entries array, usually it's a big value, an example is
> {{BlocksMap}}, this can avoid full gc since no need to resize), also we
> should be able to get Element through key.
> Following is comparison of memory footprint If we implement a lightweight set
> as described:
> We can save:
> {noformat}
> SIZE (bytes) ITEM
> 20The Key: Long (12 bytes object overhead + 8
> bytes long)
> 12HashMap Entry object overhead
> 4 reference to the key in Entry
> 4 reference to the value in Entry
> 4 hash in Entry
> {noformat}
> Total: -44 bytes
> We need to add:
> {noformat}
> SIZE (bytes) ITEM
> 4 a reference to next element in ReplicaInfo
> {noformat}
> Total: +4 bytes
> So totally we can save 40bytes for each block replica
> And currently one finalized replica needs around 46 bytes (notice: we ignore
> memory alignment here).
> We can save 1 - (4 + 46) / (44 + 46) = *45%* memory for each block replica
> in DataNode.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8857) Erasure Coding: Fix ArrayIndexOutOfBoundsException in TestWriteStripedFileWithFailure
[ https://issues.apache.org/jira/browse/HDFS-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659334#comment-14659334 ] Kai Zheng commented on HDFS-8857: - +1. > Erasure Coding: Fix ArrayIndexOutOfBoundsException in > TestWriteStripedFileWithFailure > - > > Key: HDFS-8857 > URL: https://issues.apache.org/jira/browse/HDFS-8857 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > Attachments: HDFS-8857-HDFS-7285-001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8857) Erasure Coding: Fix ArrayIndexOutOfBoundsException in TestWriteStripedFileWithFailure
[ https://issues.apache.org/jira/browse/HDFS-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659335#comment-14659335 ] Kai Zheng commented on HDFS-8857: - +1. > Erasure Coding: Fix ArrayIndexOutOfBoundsException in > TestWriteStripedFileWithFailure > - > > Key: HDFS-8857 > URL: https://issues.apache.org/jira/browse/HDFS-8857 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > Attachments: HDFS-8857-HDFS-7285-001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-8859) Improve DataNode (ReplicaMap) memory footprint to save about 45%
[
https://issues.apache.org/jira/browse/HDFS-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-8859 started by Yi Liu.
> Improve DataNode (ReplicaMap) memory footprint to save about 45%
>
>
> Key: HDFS-8859
> URL: https://issues.apache.org/jira/browse/HDFS-8859
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Critical
>
> By using following approach we can save about *45%* memory footprint for each
> block replica in DataNode memory (This JIRA only talks about *ReplicaMap* in
> DataNode), the details are:
> In ReplicaMap,
> {code}
> private final Map> map =
> new HashMap>();
> {code}
> Currently we use a HashMap {{Map}} to store the replicas
> in memory. The key is block id of the block replica which is already
> included in {{ReplicaInfo}}, so this memory can be saved. Also HashMap Entry
> has a object overhead. We can implement a lightweight Set which is similar
> to {{LightWeightGSet}}, but not a fixed size ({{LightWeightGSet}} uses fix
> size for the entries array, usually it's a big value, an example is
> {{BlocksMap}}, this can avoid full gc since no need to resize), also we
> should be able to get Element through key.
> Following is comparison of memory footprint If we implement a lightweight set
> as described:
> We can save:
> {noformat}
> SIZE (bytes) ITEM
> 20The Key: Long (12 bytes object overhead + 8
> bytes long)
> 12HashMap Entry object overhead
> 4 reference to the key in Entry
> 4 reference to the value in Entry
> 4 hash in Entry
> {noformat}
> Total: -44 bytes
> We need to add:
> {noformat}
> SIZE (bytes) ITEM
> 4 a reference to next element in ReplicaInfo
> {noformat}
> Total: +4 bytes
> So totally we can save 40bytes for each block replica
> And currently one finalized replica needs around 46 bytes (notice: we ignore
> memory alignment here).
> We can save 1 - (4 + 46) / (44 + 46) = *45%* memory for each block replica
> in DataNode.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
