[ https://issues.apache.org/jira/browse/HBASE-16458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16607737#comment-16607737 ]
Ted Yu commented on HBASE-16458: -------------------------------- On Linux, with patch, from first test output: {code} 2018-09-07 22:06:50,491 INFO [Time-limited test] hbase.ResourceChecker(148): before: backup.TestBackupUtils#TestGetBulkOutputDir Thread=8, OpenFileDescriptor=179, MaxFileDescriptor=32000, SystemLoadAverage=242, ProcessCount=363, AvailableMemoryMB=56614 {code} to last: {code} 2018-09-07 22:23:48,010 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36058 is added to blk_1073741829_1005{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-453ccfd4-ec24-490b-a51e-2b75f5b1da9f:NORMAL:127.0.0.1:36058|RBW]]} size 146414 2018-09-07 22:23:48,413 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. {code} That was ~17 minutes. > Shorten backup / restore test execution time > -------------------------------------------- > > Key: HBASE-16458 > URL: https://issues.apache.org/jira/browse/HBASE-16458 > Project: HBase > Issue Type: Test > Reporter: Ted Yu > Assignee: Vladimir Rodionov > Priority: Major > Labels: backup > Attachments: 16458.HBASE-7912.v3.txt, 16458.HBASE-7912.v4.txt, > 16458.HBASE-7912.v5.txt, 16458.v1.txt, 16458.v2.txt, 16458.v3.txt, > HBASE-16458-v1.patch > > > Below was timing information for all the backup / restore tests (today's > result): > {code} > Running org.apache.hadoop.hbase.backup.TestIncrementalBackup > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 576.273 sec - > in org.apache.hadoop.hbase.backup.TestIncrementalBackup > Running org.apache.hadoop.hbase.backup.TestBackupBoundaryTests > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 124.67 sec - > in org.apache.hadoop.hbase.backup.TestBackupBoundaryTests > Running org.apache.hadoop.hbase.backup.TestBackupStatusProgress > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.34 sec - > in org.apache.hadoop.hbase.backup.TestBackupStatusProgress > Running org.apache.hadoop.hbase.backup.TestBackupAdmin > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 490.251 sec - > in org.apache.hadoop.hbase.backup.TestBackupAdmin > Running org.apache.hadoop.hbase.backup.TestHFileArchiving > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.323 sec - > in org.apache.hadoop.hbase.backup.TestHFileArchiving > Running org.apache.hadoop.hbase.backup.TestSystemTableSnapshot > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.492 sec - > in org.apache.hadoop.hbase.backup.TestSystemTableSnapshot > Running org.apache.hadoop.hbase.backup.TestBackupDescribe > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.758 sec - > in org.apache.hadoop.hbase.backup.TestBackupDescribe > Running org.apache.hadoop.hbase.backup.TestBackupLogCleaner > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 109.187 sec - > in org.apache.hadoop.hbase.backup.TestBackupLogCleaner > Running org.apache.hadoop.hbase.backup.TestIncrementalBackupNoDataLoss > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 330.539 sec - > in org.apache.hadoop.hbase.backup.TestIncrementalBackupNoDataLoss > Running org.apache.hadoop.hbase.backup.TestRemoteBackup > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.371 sec - > in org.apache.hadoop.hbase.backup.TestRemoteBackup > Running org.apache.hadoop.hbase.backup.TestBackupSystemTable > Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.893 sec - > in org.apache.hadoop.hbase.backup.TestBackupSystemTable > Running org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.779 sec - > in org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests > Running org.apache.hadoop.hbase.backup.TestFullBackupSetRestoreSet > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 117.815 sec - > in org.apache.hadoop.hbase.backup.TestFullBackupSetRestoreSet > Running org.apache.hadoop.hbase.backup.TestBackupShowHistory > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.517 sec - > in org.apache.hadoop.hbase.backup.TestBackupShowHistory > Running org.apache.hadoop.hbase.backup.TestRemoteRestore > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.799 sec - > in org.apache.hadoop.hbase.backup.TestRemoteRestore > Running org.apache.hadoop.hbase.backup.TestFullRestore > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 317.711 sec > - in org.apache.hadoop.hbase.backup.TestFullRestore > Running org.apache.hadoop.hbase.backup.TestFullBackupSet > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.045 sec - > in org.apache.hadoop.hbase.backup.TestFullBackupSet > Running org.apache.hadoop.hbase.backup.TestBackupDelete > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.214 sec - > in org.apache.hadoop.hbase.backup.TestBackupDelete > Running org.apache.hadoop.hbase.backup.TestBackupDeleteRestore > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.631 sec - > in org.apache.hadoop.hbase.backup.TestBackupDeleteRestore > Running org.apache.hadoop.hbase.backup.TestIncrementalBackupDeleteTable > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 190.358 sec - > in org.apache.hadoop.hbase.backup.TestIncrementalBackupDeleteTable > Running > org.apache.hadoop.hbase.backup.TestBackupShowHistoryFromBackupDestination > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.726 sec - > in org.apache.hadoop.hbase.backup.TestBackupShowHistoryFromBackupDestination > Running org.apache.hadoop.hbase.backup.TestFullBackup > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 177.137 sec - > in org.apache.hadoop.hbase.backup.TestFullBackup > {code} > The duration was close to 1 hour: > {code} > [INFO] Apache HBase - Server .............................. SUCCESS [ 01:02 > h] > {code} > We should: > * merge duplicate backup / restore test cases > * shorten backup / restore test execution time (ingesting less data, etc) > * use multi wal for selected test(s) since rolling / replaying more WAL files > takes more time -- This message was sent by Atlassian JIRA (v7.6.3#76005)