[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187137#comment-16187137 ]
Hadoop QA commented on HDFS-8865: --------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} branch-2.7 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 44s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_144 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_151 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 32s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 3s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_144 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_151 {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 612 unchanged - 3 fixed = 613 total (was 615) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 60 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}683m 58s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_151. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 13m 41s{color} | {color:red} The patch generated 105 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}783m 16s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_144 Failed junit tests | hadoop.hdfs.server.blockmanagement.TestNodeCount | | | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | | | hadoop.hdfs.server.namenode.TestDeadDatanode | | | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes | | JDK v1.8.0_144 Timed out junit tests | org.apache.hadoop.hdfs.web.TestWebHdfsTokens | | JDK v1.7.0_151 Failed junit tests | hadoop.hdfs.server.namenode.TestSaveNamespace | | | hadoop.hdfs.server.namenode.snapshot.TestSnapshotManager | | | hadoop.hdfs.server.namenode.TestMetadataVersionOutput | | | hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork | | | hadoop.hdfs.TestMissingBlocksAlert | | | hadoop.hdfs.security.TestDelegationTokenForProxyUser | | | hadoop.hdfs.server.namenode.TestSecurityTokenEditLog | | | hadoop.hdfs.server.namenode.TestNameNodeOptionParsing | | | hadoop.hdfs.server.namenode.TestGetBlockLocations | | | hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement | | | hadoop.hdfs.server.namenode.TestCreateEditsLog | | JDK v1.7.0_151 Timed out junit tests | org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend | | | org.apache.hadoop.hdfs.TestEncryptionZones | | | org.apache.hadoop.hdfs.TestLeaseRecovery2 | | | org.apache.hadoop.hdfs.server.namenode.TestStartup | | | org.apache.hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages | | | org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA | | | org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing | | | org.apache.hadoop.hdfs.TestFileCreationEmpty | | | org.apache.hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy | | | org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade | | | org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager | | | org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer | | | org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer | | | org.apache.hadoop.hdfs.TestDatanodeRegistration | | | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager | | | org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints | | | org.apache.hadoop.hdfs.server.namenode.TestNameNodeHttpServer | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport | | | org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl | | | org.apache.hadoop.hdfs.TestDFSClientFailover | | | org.apache.hadoop.hdfs.TestSetrepDecreasing | | | org.apache.hadoop.hdfs.TestReservedRawPaths | | | org.apache.hadoop.hdfs.server.namenode.TestNameNodeRecovery | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength | | | org.apache.hadoop.hdfs.server.namenode.TestAddBlockRetry | | | org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure | | | org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport | | | org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus | | | org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM | | | org.apache.hadoop.hdfs.server.namenode.TestStorageRestore | | | org.apache.hadoop.hdfs.server.namenode.TestLeaseManager | | | org.apache.hadoop.hdfs.qjournal.server.TestJournalNode | | | org.apache.hadoop.hdfs.TestQuota | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot | | | org.apache.hadoop.hdfs.server.namenode.TestINodeFile | | | org.apache.hadoop.hdfs.server.namenode.TestEditLog | | | org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl | | | org.apache.hadoop.hdfs.TestDataTransferKeepalive | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot | | | org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer | | | org.apache.hadoop.hdfs.TestDatanodeDeath | | | org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings | | | org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart | | | org.apache.hadoop.hdfs.TestFileAppend | | | org.apache.hadoop.hdfs.TestPread | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot | | | org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark | | | org.apache.hadoop.hdfs.server.namenode.TestBackupNode | | | org.apache.hadoop.hdfs.server.namenode.TestFSImage | | | org.apache.hadoop.hdfs.TestSafeMode | | | org.apache.hadoop.hdfs.server.namenode.TestQuotaByStorageType | | | org.apache.hadoop.hdfs.TestFsShellPermission | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot | | | org.apache.hadoop.hdfs.TestFileAppend4 | | | org.apache.hadoop.hdfs.TestDFSFinalize | | | org.apache.hadoop.hdfs.server.namenode.TestStreamFile | | | org.apache.hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys | | | org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot | | | org.apache.hadoop.hdfs.TestRead | | | org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults | | | org.apache.hadoop.hdfs.TestDFSUpgradeFromImage | | | org.apache.hadoop.hdfs.server.namenode.TestFileContextXAttr | | | org.apache.hadoop.hdfs.server.namenode.TestFSImageWithXAttr | | | org.apache.hadoop.hdfs.TestDFSPermission | | | org.apache.hadoop.hdfs.web.TestWebHdfsTokens | | | org.apache.hadoop.hdfs.server.namenode.TestFSNamesystemMBean | | | org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives | | | org.apache.hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs | | | org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot | | | org.apache.hadoop.hdfs.security.TestDelegationToken | | | org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap | | | org.apache.hadoop.hdfs.server.namenode.TestFileLimit | | | org.apache.hadoop.hdfs.TestFileCorruption | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | | | org.apache.hadoop.hdfs.server.namenode.TestFsck | | | org.apache.hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled | | | org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits | | | org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing | | | org.apache.hadoop.hdfs.TestCrcCorruption | | | org.apache.hadoop.hdfs.TestFileCreationDelete | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters | | | org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS | | | org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer | | | org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock | | | org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits | | | org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM | | | org.apache.hadoop.hdfs.TestBlockStoragePolicy | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename | | | org.apache.hadoop.hdfs.server.namenode.TestFileTruncate | | | org.apache.hadoop.hdfs.TestRollingUpgrade | | | org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality | | | org.apache.hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate | | | org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks | | | org.apache.hadoop.hdfs.server.namenode.TestTruncateQuotaUpdate | | | org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode | | | org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures | | | org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks | | | org.apache.hadoop.hdfs.TestDFSMkdirs | | | org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite | | | org.apache.hadoop.hdfs.TestLease | | | org.apache.hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes | | | org.apache.hadoop.hdfs.server.namenode.TestMetaSave | | | org.apache.hadoop.hdfs.qjournal.TestNNWithQJM | | | org.apache.hadoop.hdfs.web.TestWebHDFS | | | org.apache.hadoop.hdfs.server.namenode.TestCommitBlockWithInvalidGenStamp | | | org.apache.hadoop.hdfs.server.namenode.TestINodeAttributeProvider | | | org.apache.hadoop.hdfs.server.namenode.TestCheckpoint | | | org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean | | | org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster | | | org.apache.hadoop.hdfs.tools.TestGetGroups | | | org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount | | | org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs | | | org.apache.hadoop.hdfs.web.TestWebHDFSXAttr | | | org.apache.hadoop.hdfs.TestRollingUpgradeRollback | | | org.apache.hadoop.hdfs.TestBlockReaderLocal | | | org.apache.hadoop.hdfs.server.namenode.ha.TestDFSZKFailoverController | | | org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache | | | org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes | | | org.apache.hadoop.hdfs.server.namenode.TestFSDirectory | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication | | | org.apache.hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA | | | org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | | | org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup | | | org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication | | | org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader | | | org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache | | | org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication | | | org.apache.hadoop.hdfs.server.namenode.TestFSImageWithAcl | | | org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions | | | org.apache.hadoop.hdfs.server.namenode.TestXAttrConfigFlag | | | org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead | | | org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat | | | org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover | | | org.apache.hadoop.hdfs.server.namenode.TestDeleteRace | | | org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA | | | org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby | | | org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA | | | org.apache.hadoop.hdfs.server.namenode.TestEditLogRace | | | org.apache.hadoop.hdfs.server.namenode.TestAddBlock | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion | | | org.apache.hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA | | | org.apache.hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd | | | org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs | | | org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction | | | org.apache.hadoop.hdfs.TestBlockReaderFactory | | | org.apache.hadoop.hdfs.TestDistributedFileSystem | | | org.apache.hadoop.hdfs.server.blockmanagement.TestSequentialBlockId | | | org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | | org.apache.hadoop.hdfs.web.TestWebHDFSForHA | | | org.apache.hadoop.hdfs.server.namenode.ha.TestQuotasWithHA | | | org.apache.hadoop.hdfs.TestEncryptedTransfer | | | org.apache.hadoop.hdfs.TestDFSClientExcludedNodes | | | org.apache.hadoop.hdfs.server.namenode.TestAuditLogs | | | org.apache.hadoop.hdfs.TestDFSShell | | | org.apache.hadoop.hdfs.server.namenode.TestHostsFiles | | | org.apache.hadoop.hdfs.server.namenode.TestAuditLogger | | | org.apache.hadoop.hdfs.TestPersistBlocks | | | org.apache.hadoop.hdfs.tools.TestDebugAdmin | | | org.apache.hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics | | | org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot | | | org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode | | | org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens | | | org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | | | org.apache.hadoop.hdfs.TestGetFileChecksum | | | org.apache.hadoop.hdfs.web.TestWebHDFSAcl | | | org.apache.hadoop.hdfs.TestAbandonBlock | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:67e87c9 | | JIRA Issue | HDFS-8865 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12889763/HDFS-8865.branch-2.7.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux e18536e63d2c 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | branch-2.7 / 90aae39 | | Default Java | 1.7.0_151 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/21463/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/21463/artifact/patchprocess/whitespace-eol.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_151.txt | | JDK v1.7.0_151 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21463/testReport/ | | asflicense | https://builds.apache.org/job/PreCommit-HDFS-Build/21463/artifact/patchprocess/patch-asflicense-problems.txt | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21463/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Improve quota initialization performance > ---------------------------------------- > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Kihwal Lee > Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org