[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497961#comment-16497961 ] Kihwal Lee commented on HDFS-8865: -- The memory overhead is very small compared to the memory required to hold the name space. The creation of child tasks is somwwhat throttled by the threadpool size. I.e. it does not explode walking entire namespace at once. Be sure to port HDFS-9003 with it. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Major > Fix For: 2.8.0, 3.0.0-alpha1, 2.6.6, 2.7.5 > > Attachments: HDFS-8865.branch-2.6.01.patch, > HDFS-8865.branch-2.6.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch, > HDFS-8865_branch-2.6.patch, HDFS-8865_branch-2.7.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497816#comment-16497816 ] Gang Xie commented on HDFS-8865: We want to port this path to our 2.4 release. But we have a little concern about the mem pressure after multi-threaded, especially when it starts and transition to active state immediately, I.E. after the full block report. Do we have any mem pressure test result about this? One idea to optimize this is not to held the children list since the namespace is not changed during this update. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Major > Fix For: 2.8.0, 3.0.0-alpha1, 2.6.6, 2.7.5 > > Attachments: HDFS-8865.branch-2.6.01.patch, > HDFS-8865.branch-2.6.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch, > HDFS-8865_branch-2.6.patch, HDFS-8865_branch-2.7.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188983#comment-16188983 ] Yongjun Zhang commented on HDFS-8865: - Thanks [~xiaochen], +1 on the updated 2.6 patch (01). > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865.branch-2.6.01.patch, > HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188666#comment-16188666 ] Yongjun Zhang commented on HDFS-8865: - Thanks [~xiaochen] for working on this. +1 on the new revs for 2.7. Question on 2.6: Good to add the test to the 2.6 new patch. There is only one new test for this jira in 2.7 and beyond. Though it helps testing by including additional tests, should the other tests come with backporting different jiras (HDFS-10843 etc)? Minor: when uploading new patches for the same branch in the future, suggest to include a version string. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187671#comment-16187671
]
Xiao Chen commented on HDFS-8865:
-
I gave up on pre-commits and went on to run hadoop-hdfs tests in internal
infra. Results below:
{noformat}
branch-2.7:
Test Result (12 failures / +12)
org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten
org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner.testThrottling
org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode.testNonDFSUsedONDeadNodeReReg
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testSnapshotWithAppendTruncate
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testCopyOnTruncateWithDataNodesRestart
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testSnapshotWithTruncates
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncateRecovery
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncateShellCommandOnBlockBoundary
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncateWithDataNodesRestartImmediately
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncate4Symlink
branch-2.6:
Test Result (7 failures / -5)
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testBasicFunctionality
org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner.testDataBlockScanner
org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner.testBlockScannerAfterRefresh
org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner.testBlockScannerAfterRestart
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles.testFallbackToDiskFull
org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache.testDataXceiverCleansUpSlotsOnFailure
{noformat}
All passed locally, except {{TestDeadDatanode#testNonDFSUsedONDeadNodeReReg}}
for branch-2.7 and {{TestLazyPersistFiles#testFallbackToDiskFull}} for
branch-2.6 are failing with or without this patch.
Will file a jira for branch-2.7, and backport the following (in the exact
order, left is earlier) HDFS-9072->HDFS-9740->HDFS-10270->HDFS-9743 to fix
branch-2.6.
Plan to commit this by end of Monday if no objections.
Thanks.
> Improve quota initialization performance
>
>
> Key: HDFS-8865
> URL: https://issues.apache.org/jira/browse/HDFS-8865
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch,
> HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch,
> HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch
>
>
> After replaying edits, the whole file system tree is recursively scanned in
> order to initialize the quota. For big name space, this can take a very long
> time. Since this is done during namenode failover, it also affects failover
> latency.
> By using the Fork-Join framework, I was able to greatly reduce the
> initialization time. The following is the test result using the fsimage from
> one of the big name nodes we have.
> || threads || seconds||
> | 1 (existing) | 55|
> | 1 (fork-join) | 68 |
> | 4 | 16 |
> | 8 | 8 |
> | 12 | 6 |
> | 16 | 5 |
> | 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187137#comment-16187137
]
Hadoop QA commented on HDFS-8865:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.7 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_151 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 612 unchanged - 3 fixed = 613 total (was 615) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 60 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}683m 58s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_151. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 13m
41s{color} | {color:red} The patch generated 105 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}783m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_144 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.server.namenode.TestDeadDatanode |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| JDK v1.8.0_144 Timed out junit tests |
org.apache.hadoop.hdfs.web.TestWebHdfsTokens |
| JDK v1.7.0_151 Failed junit tests |
hadoop.hdfs.server.namenode.TestSaveNamespace |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotManager |
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187012#comment-16187012
]
Hadoop QA commented on HDFS-8865:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.7 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_151 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 611 unchanged - 3 fixed = 612 total (was 614) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 60 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}696m 51s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_151. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 12m
48s{color} | {color:red} The patch generated 57 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}801m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_144 Failed junit tests | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.server.namenode.TestDeadDatanode |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.web.TestWebHdfsTokens |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| JDK v1.7.0_151 Failed junit tests |
hadoop.hdfs.server.namenode.TestLeaseManager |
| | hadoop.hdfs.server.datanode.TestDataNodeInitStorage |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| |
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186987#comment-16186987
]
Hadoop QA commented on HDFS-8865:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.7 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_151 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 611 unchanged - 3 fixed = 612 total (was 614) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 60 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}537m 46s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_151. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 6m
24s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}623m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_144 Failed junit tests |
hadoop.hdfs.server.namenode.TestDeadDatanode |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.web.TestHttpsFileSystem |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| JDK v1.7.0_151 Failed junit tests | hadoop.hdfs.TestRead |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| |
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186398#comment-16186398 ] Xiao Chen commented on HDFS-8865: - INFRA-15194 is failing precommits... Trying to make 2.7 backport executed first. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186011#comment-16186011 ] Xiao Chen commented on HDFS-8865: - hm, https://builds.apache.org/job/PreCommit-hdfs-Build/21423/ seems to confused the branch-2.7 patch and branch-2.6 patch. I see messages saying both of them getting applied... Removing 2.6 and retriggered, hopefully faster than looking at why yetus failed. bq. Xiao Chen would you please also add TestDiskspaceQuotaUpdate into the 2.6 patch? IIRC I tried very briefly, but need other changes to make it compile, so stopped there. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865_branch-2.7.patch, > HDFS-8865.branch-2.7.patch, HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185374#comment-16185374 ] Wei-Chiu Chuang commented on HDFS-8865: --- [~xiaochen] would you please also add TestDiskspaceQuotaUpdate into the 2.6 patch? > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185288#comment-16185288 ] Xiao Chen commented on HDFS-8865: - Attached 2.7 patch, which is backported from the branch-2.8 commit. Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865_branch-2.7.patch, > HDFS-8865.branch-2.7.patch, HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504488#comment-15504488 ] Kihwal Lee commented on HDFS-8865: -- Just added you as a HDFS contributor. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch, HDFS-8865_branch-2.7.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503952#comment-15503952 ] Nicolas Fraison commented on HDFS-8865: --- [~kihwal] I would appreciate to be added as a contributor. Even if it is not pushed to any release the backport will be available here for anyone needing it. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch, HDFS-8865_branch-2.7.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503897#comment-15503897 ] Kihwal Lee commented on HDFS-8865: -- [~nfraison.criteo], if you plan to contribute code to the HDFS project in the future, I can add you as a contributor. That will allow you to submit patches. If you want to be a contributor, just let me know. As for porting this specific jira to branch-2.6, the release branch for 2.6.5 was just cut, so it will not make the next release. It is also unclear whether 2.6.6 will ever be released. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch, HDFS-8865_branch-2.7.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503755#comment-15503755 ] Nicolas Fraison commented on HDFS-8865: --- We have backported this patch to our 2.6 builds. Could you please let me knwo how I can provide it, I don't have any rights to attach the git diff file. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch, HDFS-8865_branch-2.7.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720344#comment-14720344 ] Daryn Sharp commented on HDFS-8865: --- +1 This has made a huge difference, and all the possible style warning were addressed. Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720363#comment-14720363 ] Kihwal Lee commented on HDFS-8865: -- Thanks [~xyao] and [~daryn] for reviews. I've committed this to trunk and branch-2. Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720368#comment-14720368 ] Hudson commented on HDFS-8865: -- FAILURE: Integrated in Hadoop-trunk-Commit #8364 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8364/]) HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720620#comment-14720620 ] Hudson commented on HDFS-8865: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2247 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2247/]) HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720628#comment-14720628 ] Hudson commented on HDFS-8865: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #309 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/309/]) HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720580#comment-14720580 ] Hudson commented on HDFS-8865: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2266 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2266/]) HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720495#comment-14720495 ] Hudson commented on HDFS-8865: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #322 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/322/]) HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720452#comment-14720452 ] Hudson commented on HDFS-8865: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #317 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/317/]) HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720448#comment-14720448 ] Hudson commented on HDFS-8865: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1050 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1050/]) HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715267#comment-14715267 ] Kihwal Lee commented on HDFS-8865: -- - The checkstyle warning is against DFSConfigKeys. - TestEncryptedTransfer failure is not related this patch. Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715624#comment-14715624
]
Hadoop QA commented on HDFS-8865:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 48s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 52s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 23s | The applied patch generated 1
new checkstyle issues (total was 471, now 469). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 188m 5s | Tests failed in hadoop-hdfs. |
| | | 233m 10s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFSNamesystem |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751597/HDFS-8865.v3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a4d9acc |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12140/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12140/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12140/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12140/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12140/console |
This message was automatically generated.
Improve quota initialization performance
Key: HDFS-8865
URL: https://issues.apache.org/jira/browse/HDFS-8865
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch,
HDFS-8865.v2.patch, HDFS-8865.v3.patch
After replaying edits, the whole file system tree is recursively scanned in
order to initialize the quota. For big name space, this can take a very long
time. Since this is done during namenode failover, it also affects failover
latency.
By using the Fork-Join framework, I was able to greatly reduce the
initialization time. The following is the test result using the fsimage from
one of the big name nodes we have.
|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707281#comment-14707281
]
Hadoop QA commented on HDFS-8865:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 50s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 9m 22s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 27s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 42s | The applied patch generated 1
new checkstyle issues (total was 471, now 469). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 55s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 46s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 15s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 42s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 80m 10s | Tests failed in hadoop-hdfs. |
| | | 131m 57s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | org.apache.hadoop.hdfs.TestEncryptedTransfer |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751597/HDFS-8865.v3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 22de7c1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12067/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12067/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12067/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12067/console |
This message was automatically generated.
Improve quota initialization performance
Key: HDFS-8865
URL: https://issues.apache.org/jira/browse/HDFS-8865
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch,
HDFS-8865.v2.patch, HDFS-8865.v3.patch
After replaying edits, the whole file system tree is recursively scanned in
order to initialize the quota. For big name space, this can take a very long
time. Since this is done during namenode failover, it also affects failover
latency.
By using the Fork-Join framework, I was able to greatly reduce the
initialization time. The following is the test result using the fsimage from
one of the big name nodes we have.
|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705828#comment-14705828
]
Kihwal Lee commented on HDFS-8865:
--
Thanks for the review [~xyao].
1. Numbers for small/medium size name space: I do not have numbers for them,
but I expect them to be better than before. The test cases show it taking a
couple of milliseconds for a very small name space, so the overhead of
fork-join doesn't seem to be of too much concern. For very small namespaces,
the ones that take far less than a second to initialize, the multithreaded
initialization cpuld take a bit longer.
2. This involves conversion to slf4j. I started down this path, but ended up
needing to update {{IOUtils}} as well. Since it is not a matter of updating
just {{FSImage}}, I won't do it in this jira. Instead I will simply wrap it
with {{isDebugEnabled()}} check for now.
Improve quota initialization performance
Key: HDFS-8865
URL: https://issues.apache.org/jira/browse/HDFS-8865
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch,
HDFS-8865.v2.patch
After replaying edits, the whole file system tree is recursively scanned in
order to initialize the quota. For big name space, this can take a very long
time. Since this is done during namenode failover, it also affects failover
latency.
By using the Fork-Join framework, I was able to greatly reduce the
initialization time. The following is the test result using the fsimage from
one of the big name nodes we have.
|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695574#comment-14695574
]
Xiaoyu Yao commented on HDFS-8865:
--
Thanks for the patch, [~kihwal]! It looks pretty good to me.
Just a few comments:
1. The number for large namespace looks impressive. Do you have the number for
small/medium namespace?
2. Is it possible to add some profiling info between these logs below so that
we can easily find how long it takes to finish quota initialization from the
log?
{code}
LOG.info(Initializing quota with + threads + thread(s));
...
LOG.info(Quota initialization complete.\n + counts);
{code}
3. Can you change to parameterized logging to avoid parameter construction in
case the log statement is disabled. For example,
{code}
LOG.debug(Setting quota for {} +\n{}, dir, myCounts);
{code}
4. NIT: typo chached - cached?
{code}
// Directly access the name system to obtain the current chached usage.
{code}
5. Now that HDFS-8879 is in, can you rebase and update the patch? Thanks!
Improve quota initialization performance
Key: HDFS-8865
URL: https://issues.apache.org/jira/browse/HDFS-8865
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch,
HDFS-8865.v2.patch
After replaying edits, the whole file system tree is recursively scanned in
order to initialize the quota. For big name space, this can take a very long
time. Since this is done during namenode failover, it also affects failover
latency.
By using the Fork-Join framework, I was able to greatly reduce the
initialization time. The following is the test result using the fsimage from
one of the big name nodes we have.
|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14682301#comment-14682301
]
Hadoop QA commented on HDFS-8865:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 9s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 41s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 56s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 21s | The applied patch generated 1
new checkstyle issues (total was 493, now 491). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 21s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 38s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 186m 4s | Tests failed in hadoop-hdfs. |
| | | 230m 19s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749856/HDFS-8865.v2.checkstyle.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fa1d84a |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11962/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11962/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11962/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11962/console |
This message was automatically generated.
Improve quota initialization performance
Key: HDFS-8865
URL: https://issues.apache.org/jira/browse/HDFS-8865
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch,
HDFS-8865.v2.patch
After replaying edits, the whole file system tree is recursively scanned in
order to initialize the quota. For big name space, this can take a very long
time. Since this is done during namenode failover, it also affects failover
latency.
By using the Fork-Join framework, I was able to greatly reduce the
initialization time. The following is the test result using the fsimage from
one of the big name nodes we have.
|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14682371#comment-14682371 ] Kihwal Lee commented on HDFS-8865: -- The checkstyle error is for the new config key, which I am not going to fix. The unit test timeout does not happen when I run it. Looks like it is failing in other pre-commit builds too, so it is not being caused by this patch. Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680185#comment-14680185 ] Xiaoyu Yao commented on HDFS-8865: -- Thanks [~kihwal]! I agree with your comments above and post a patch on HDFS-8879. This will help us to backport these fixes to 2.7 branches. Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch, HDFS-8865.v2.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680374#comment-14680374
]
Hadoop QA commented on HDFS-8865:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 40s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 42s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 21s | The applied patch generated 3
new checkstyle issues (total was 491, now 491). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 20s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 4s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 176m 53s | Tests failed in hadoop-hdfs. |
| | | 221m 11s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749585/HDFS-8865.v2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11951/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11951/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11951/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11951/console |
This message was automatically generated.
Improve quota initialization performance
Key: HDFS-8865
URL: https://issues.apache.org/jira/browse/HDFS-8865
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8865.patch, HDFS-8865.v2.patch
After replaying edits, the whole file system tree is recursively scanned in
order to initialize the quota. For big name space, this can take a very long
time. Since this is done during namenode failover, it also affects failover
latency.
By using the Fork-Join framework, I was able to greatly reduce the
initialization time. The following is the test result using the fsimage from
one of the big name nodes we have.
|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680076#comment-14680076
]
Kihwal Lee commented on HDFS-8865:
--
- Fixed the findbug warning of unused variable.
- Fixed the test breakages. Snapshot tests were breaking because of casting ref
inode to dir. Changed it to use {{child.asDirectory()}} when generating work.
{{TestLeaseRecovery2}} and {{TestHDFSCLI}} are also passing locally.
- Addressed most checkstyle issues. A couple of them are ignored.
Improve quota initialization performance
Key: HDFS-8865
URL: https://issues.apache.org/jira/browse/HDFS-8865
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8865.patch, HDFS-8865.v2.patch
After replaying edits, the whole file system tree is recursively scanned in
order to initialize the quota. For big name space, this can take a very long
time. Since this is done during namenode failover, it also affects failover
latency.
By using the Fork-Join framework, I was able to greatly reduce the
initialization time. The following is the test result using the fsimage from
one of the big name nodes we have.
|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680089#comment-14680089 ] Kihwal Lee commented on HDFS-8865: -- bq. Can we fix the quota by storage type update issue in a separate JIRA? It is kind of hard to not do it, since that part of code is being modified. I will just do whatever seems natural in the new code. We can make sure it is fixed properly and improve the test case in HDFS-8879. Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch, HDFS-8865.v2.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662585#comment-14662585 ] Xiaoyu Yao commented on HDFS-8865: -- [~kihwal], thanks for working on this improvement work and fixing the issue on quota by storage type usage update. Can we fix the quota by storage type update issue in a separate JIRA? Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662762#comment-14662762
]
Hadoop QA commented on HDFS-8865:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 26m 11s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 59s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 20s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 55s | The applied patch generated
10 new checkstyle issues (total was 491, now 498). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 44s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 9s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 175m 32s | Tests failed in hadoop-hdfs. |
| | | 231m 23s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749347/HDFS-8865.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11938/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11938/console |
This message was automatically generated.
Improve quota initialization performance
Key: HDFS-8865
URL: https://issues.apache.org/jira/browse/HDFS-8865
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8865.patch
After replaying edits, the whole file system tree is recursively scanned in
order to initialize the quota. For big name space, this can take a very long
time. Since this is done during namenode failover, it also affects failover
latency.
By using the Fork-Join framework, I was able to greatly reduce the
initialization time. The following is the test result using the fsimage from
one of the big name nodes we have.
|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660904#comment-14660904
]
Kihwal Lee commented on HDFS-8865:
--
While porting the patch from 2.6 to trunk, I noticed something odd in
{{FSImagte.java}}.
{code}
final EnumCountersStorageType typeSpaces =
new EnumCountersStorageType(StorageType.class);
for (StorageType t : StorageType.getTypesSupportingQuota()) {
final long typeSpace = counts.getTypeSpaces().get(t) -
parentTypeSpaces.get(t);
final long typeQuota = q.getTypeSpaces().get(t);
if (Quota.isViolated(typeQuota, typeSpace)) {
LOG.warn(Storage type quota violation in image for
+ dir.getFullPathName()
+ type = + t.toString() + quota =
+ typeQuota + consumed + typeSpace);
}
}
dir.getDirectoryWithQuotaFeature().setSpaceConsumed(namespace, ssConsumed,
typeSpaces);
{code}
{{typeSpaces}} is passed to {{setSpaceConsumed}}, but it is empty. On restart
of NN, storage type-based quota usage tracking will be reset.
Improve quota initialization performance
Key: HDFS-8865
URL: https://issues.apache.org/jira/browse/HDFS-8865
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
After replaying edits, the whole file system tree is recursively scanned in
order to initialize the quota. For big name space, this can take a very long
time. Since this is done during namenode failover, it also affects failover
latency.
By using the Fork-Join framework, I was able to greatly reduce the
initialization time. The following is the test result using the fsimage from
one of the big name nodes we have.
|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660498#comment-14660498 ] Kihwal Lee commented on HDFS-8865: -- I will post the patch shortly. Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
