Re: Long test run times
There is probably some good amount of low hanging fruit, we can make some headway and see how it goes from there? On Thu, Jul 29, 2021 at 7:18 PM Danny Chan wrote: > What should we do for these long running tests ? Simplify them to more > simple UTs ? > > Vinoth Chandar 于2021年7月30日周五 上午6:53写道： > > > I am looking into > > > > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > > > > On Thu, Jul 29, 2021 at 1:54 PM Sivabalan wrote: > > > > > I will take up some. > > > > > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > > > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > > > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > > > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 29, 2021 at 4:04 PM Vinoth Chandar > > wrote: > > > > > > > Folks, > > > > > > > > Our tests are now exceeding 60 minutes and suffering timeouts on > azure > > > > (travis is slow to actually schedule, but seems to finish on time). > > > > > > > > Following are the list of top slow tests. I am working from the top > of > > > the > > > > list. Any one interested in chipping in? Please respond with the > test > > > you > > > > are interested in fixing. > > > > > > > > 927.371 > > > > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > > > > 894.438 > > > > > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > > > > 866.088 org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > > > > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > > > > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > > > > 291.229 org.apache.hudi.table.TestHoodieMergeOnReadTable > > > > 245.922 org.apache.hudi.functional.TestStructuredStreaming > > > > 234.737 org.apache.hudi.index.TestHoodieIndex > > > > 224.893 org.apache.hudi.functional.TestMORDataSource > > > > 217.354 > > > > > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > > > 210.717 org.apache.hudi.common.functional.TestHoodieLogFormat > > > > 204.989 org.apache.hudi.table.TestCleaner > > > > 198.623 org.apache.hudi.client.TestBootstrap > > > > 170.029 org.apache.hudi.hive.TestHiveSyncTool > > > > 135.295 org.apache.hudi.table.action.compact.TestInlineCompaction > > > > 116.381 > > org.apache.hudi.table.action.commit.TestCopyOnWriteActionExecutor > > > > 113.559 org.apache.hudi.functional.TestCOWDataSource > > > > 111.297 org.apache.hudi.table.upgrade.TestUpgradeDowngrade > > > > > > > > Thanks > > > > Vinoth > > > > > > > > > > > > > -- > > > Regards, > > > -Sivabalan > > > > > >
Re: Long test run times
What should we do for these long running tests ? Simplify them to more simple UTs ? Vinoth Chandar 于2021年7月30日周五 上午6:53写道： > I am looking into > > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > > On Thu, Jul 29, 2021 at 1:54 PM Sivabalan wrote: > > > I will take up some. > > > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > > > > > > > > > > > > > > > > > On Thu, Jul 29, 2021 at 4:04 PM Vinoth Chandar > wrote: > > > > > Folks, > > > > > > Our tests are now exceeding 60 minutes and suffering timeouts on azure > > > (travis is slow to actually schedule, but seems to finish on time). > > > > > > Following are the list of top slow tests. I am working from the top of > > the > > > list. Any one interested in chipping in? Please respond with the test > > you > > > are interested in fixing. > > > > > > 927.371 > > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > > > 894.438 > > > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > > > 866.088 org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > > > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > > > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > > > 291.229 org.apache.hudi.table.TestHoodieMergeOnReadTable > > > 245.922 org.apache.hudi.functional.TestStructuredStreaming > > > 234.737 org.apache.hudi.index.TestHoodieIndex > > > 224.893 org.apache.hudi.functional.TestMORDataSource > > > 217.354 > > > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > > 210.717 org.apache.hudi.common.functional.TestHoodieLogFormat > > > 204.989 org.apache.hudi.table.TestCleaner > > > 198.623 org.apache.hudi.client.TestBootstrap > > > 170.029 org.apache.hudi.hive.TestHiveSyncTool > > > 135.295 org.apache.hudi.table.action.compact.TestInlineCompaction > > > 116.381 > org.apache.hudi.table.action.commit.TestCopyOnWriteActionExecutor > > > 113.559 org.apache.hudi.functional.TestCOWDataSource > > > 111.297 org.apache.hudi.table.upgrade.TestUpgradeDowngrade > > > > > > Thanks > > > Vinoth > > > > > > > > > -- > > Regards, > > -Sivabalan > > >
Re: Long test run times
I am looking into 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata On Thu, Jul 29, 2021 at 1:54 PM Sivabalan wrote: > I will take up some. > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > > > > > > > > On Thu, Jul 29, 2021 at 4:04 PM Vinoth Chandar wrote: > > > Folks, > > > > Our tests are now exceeding 60 minutes and suffering timeouts on azure > > (travis is slow to actually schedule, but seems to finish on time). > > > > Following are the list of top slow tests. I am working from the top of > the > > list. Any one interested in chipping in? Please respond with the test > you > > are interested in fixing. > > > > 927.371 > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > > 894.438 > > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > > 866.088 org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > > 291.229 org.apache.hudi.table.TestHoodieMergeOnReadTable > > 245.922 org.apache.hudi.functional.TestStructuredStreaming > > 234.737 org.apache.hudi.index.TestHoodieIndex > > 224.893 org.apache.hudi.functional.TestMORDataSource > > 217.354 > > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > 210.717 org.apache.hudi.common.functional.TestHoodieLogFormat > > 204.989 org.apache.hudi.table.TestCleaner > > 198.623 org.apache.hudi.client.TestBootstrap > > 170.029 org.apache.hudi.hive.TestHiveSyncTool > > 135.295 org.apache.hudi.table.action.compact.TestInlineCompaction > > 116.381 org.apache.hudi.table.action.commit.TestCopyOnWriteActionExecutor > > 113.559 org.apache.hudi.functional.TestCOWDataSource > > 111.297 org.apache.hudi.table.upgrade.TestUpgradeDowngrade > > > > Thanks > > Vinoth > > > > > -- > Regards, > -Sivabalan >
Re: Long test run times
I will take up some. org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter On Thu, Jul 29, 2021 at 4:04 PM Vinoth Chandar wrote: > Folks, > > Our tests are now exceeding 60 minutes and suffering timeouts on azure > (travis is slow to actually schedule, but seems to finish on time). > > Following are the list of top slow tests. I am working from the top of the > list. Any one interested in chipping in? Please respond with the test you > are interested in fixing. > > 927.371 > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > 894.438 > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > 866.088 org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > 291.229 org.apache.hudi.table.TestHoodieMergeOnReadTable > 245.922 org.apache.hudi.functional.TestStructuredStreaming > 234.737 org.apache.hudi.index.TestHoodieIndex > 224.893 org.apache.hudi.functional.TestMORDataSource > 217.354 > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > 210.717 org.apache.hudi.common.functional.TestHoodieLogFormat > 204.989 org.apache.hudi.table.TestCleaner > 198.623 org.apache.hudi.client.TestBootstrap > 170.029 org.apache.hudi.hive.TestHiveSyncTool > 135.295 org.apache.hudi.table.action.compact.TestInlineCompaction > 116.381 org.apache.hudi.table.action.commit.TestCopyOnWriteActionExecutor > 113.559 org.apache.hudi.functional.TestCOWDataSource > 111.297 org.apache.hudi.table.upgrade.TestUpgradeDowngrade > > Thanks > Vinoth > -- Regards, -Sivabalan
Long test run times
Folks, Our tests are now exceeding 60 minutes and suffering timeouts on azure (travis is slow to actually schedule, but seems to finish on time). Following are the list of top slow tests. I am working from the top of the list. Any one interested in chipping in? Please respond with the test you are interested in fixing. 927.371 org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer 894.438 org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite 866.088 org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata 291.229 org.apache.hudi.table.TestHoodieMergeOnReadTable 245.922 org.apache.hudi.functional.TestStructuredStreaming 234.737 org.apache.hudi.index.TestHoodieIndex 224.893 org.apache.hudi.functional.TestMORDataSource 217.354 org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter 210.717 org.apache.hudi.common.functional.TestHoodieLogFormat 204.989 org.apache.hudi.table.TestCleaner 198.623 org.apache.hudi.client.TestBootstrap 170.029 org.apache.hudi.hive.TestHiveSyncTool 135.295 org.apache.hudi.table.action.compact.TestInlineCompaction 116.381 org.apache.hudi.table.action.commit.TestCopyOnWriteActionExecutor 113.559 org.apache.hudi.functional.TestCOWDataSource 111.297 org.apache.hudi.table.upgrade.TestUpgradeDowngrade Thanks Vinoth
