Re: Long test run times
There is probably some good amount of low hanging fruit, we can make some headway and see how it goes from there? On Thu, Jul 29, 2021 at 7:18 PM Danny Chan wrote: > What should we do for these long running tests ? Simplify them to more > simple UTs ? > > Vinoth Chandar 于2021年7月30日周五 上午6:53写道： > > > I am looking into > > > > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > > > > On Thu, Jul 29, 2021 at 1:54 PM Sivabalan wrote: > > > > > I will take up some. > > > > > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > > > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > > > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > > > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 29, 2021 at 4:04 PM Vinoth Chandar > > wrote: > > > > > > > Folks, > > > > > > > > Our tests are now exceeding 60 minutes and suffering timeouts on > azure > > > > (travis is slow to actually schedule, but seems to finish on time). > > > > > > > > Following are the list of top slow tests. I am working from the top > of > > > the > > > > list. Any one interested in chipping in? Please respond with the > test > > > you > > > > are interested in fixing. > > > > > > > > 927.371 > > > > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > > > > 894.438 > > > > > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > > > > 866.088 org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > > > > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > > > > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > > > > 291.229 org.apache.hudi.table.TestHoodieMergeOnReadTable > > > > 245.922 org.apache.hudi.functional.TestStructuredStreaming > > > > 234.737 org.apache.hudi.index.TestHoodieIndex > > > > 224.893 org.apache.hudi.functional.TestMORDataSource > > > > 217.354 > > > > > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > > > 210.717 org.apache.hudi.common.functional.TestHoodieLogFormat > > > > 204.989 org.apache.hudi.table.TestCleaner > > > > 198.623 org.apache.hudi.client.TestBootstrap > > > > 170.029 org.apache.hudi.hive.TestHiveSyncTool > > > > 135.295 org.apache.hudi.table.action.compact.TestInlineCompaction > > > > 116.381 > > org.apache.hudi.table.action.commit.TestCopyOnWriteActionExecutor > > > > 113.559 org.apache.hudi.functional.TestCOWDataSource > > > > 111.297 org.apache.hudi.table.upgrade.TestUpgradeDowngrade > > > > > > > > Thanks > > > > Vinoth > > > > > > > > > > > > > -- > > > Regards, > > > -Sivabalan > > > > > >
Re: Long test run times
What should we do for these long running tests ? Simplify them to more simple UTs ? Vinoth Chandar 于2021年7月30日周五 上午6:53写道： > I am looking into > > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > > On Thu, Jul 29, 2021 at 1:54 PM Sivabalan wrote: > > > I will take up some. > > > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > > > > > > > > > > > > > > > > > On Thu, Jul 29, 2021 at 4:04 PM Vinoth Chandar > wrote: > > > > > Folks, > > > > > > Our tests are now exceeding 60 minutes and suffering timeouts on azure > > > (travis is slow to actually schedule, but seems to finish on time). > > > > > > Following are the list of top slow tests. I am working from the top of > > the > > > list. Any one interested in chipping in? Please respond with the test > > you > > > are interested in fixing. > > > > > > 927.371 > > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > > > 894.438 > > > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > > > 866.088 org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > > > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > > > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > > > 291.229 org.apache.hudi.table.TestHoodieMergeOnReadTable > > > 245.922 org.apache.hudi.functional.TestStructuredStreaming > > > 234.737 org.apache.hudi.index.TestHoodieIndex > > > 224.893 org.apache.hudi.functional.TestMORDataSource > > > 217.354 > > > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > > 210.717 org.apache.hudi.common.functional.TestHoodieLogFormat > > > 204.989 org.apache.hudi.table.TestCleaner > > > 198.623 org.apache.hudi.client.TestBootstrap > > > 170.029 org.apache.hudi.hive.TestHiveSyncTool > > > 135.295 org.apache.hudi.table.action.compact.TestInlineCompaction > > > 116.381 > org.apache.hudi.table.action.commit.TestCopyOnWriteActionExecutor > > > 113.559 org.apache.hudi.functional.TestCOWDataSource > > > 111.297 org.apache.hudi.table.upgrade.TestUpgradeDowngrade > > > > > > Thanks > > > Vinoth > > > > > > > > > -- > > Regards, > > -Sivabalan > > >
Re: Long test run times
I am looking into 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata On Thu, Jul 29, 2021 at 1:54 PM Sivabalan wrote: > I will take up some. > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > > > > > > > > On Thu, Jul 29, 2021 at 4:04 PM Vinoth Chandar wrote: > > > Folks, > > > > Our tests are now exceeding 60 minutes and suffering timeouts on azure > > (travis is slow to actually schedule, but seems to finish on time). > > > > Following are the list of top slow tests. I am working from the top of > the > > list. Any one interested in chipping in? Please respond with the test > you > > are interested in fixing. > > > > 927.371 > > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > > 894.438 > > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > > 866.088 org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > > 291.229 org.apache.hudi.table.TestHoodieMergeOnReadTable > > 245.922 org.apache.hudi.functional.TestStructuredStreaming > > 234.737 org.apache.hudi.index.TestHoodieIndex > > 224.893 org.apache.hudi.functional.TestMORDataSource > > 217.354 > > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > > 210.717 org.apache.hudi.common.functional.TestHoodieLogFormat > > 204.989 org.apache.hudi.table.TestCleaner > > 198.623 org.apache.hudi.client.TestBootstrap > > 170.029 org.apache.hudi.hive.TestHiveSyncTool > > 135.295 org.apache.hudi.table.action.compact.TestInlineCompaction > > 116.381 org.apache.hudi.table.action.commit.TestCopyOnWriteActionExecutor > > 113.559 org.apache.hudi.functional.TestCOWDataSource > > 111.297 org.apache.hudi.table.upgrade.TestUpgradeDowngrade > > > > Thanks > > Vinoth > > > > > -- > Regards, > -Sivabalan >
Re: Long test run times
I will take up some. org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter On Thu, Jul 29, 2021 at 4:04 PM Vinoth Chandar wrote: > Folks, > > Our tests are now exceeding 60 minutes and suffering timeouts on azure > (travis is slow to actually schedule, but seems to finish on time). > > Following are the list of top slow tests. I am working from the top of the > list. Any one interested in chipping in? Please respond with the test you > are interested in fixing. > > 927.371 > org.apache.hudi.utilities.functional.TestHoodieMultiTableDeltaStreamer > 894.438 > org.apache.hudi.spark3.internal.TestHoodieDataSourceInternalBatchWrite > 866.088 org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer > 614.322 org.apache.hudi.client.TestHoodieClientOnCopyOnWriteStorage > 556.392 org.apache.hudi.metadata.TestHoodieBackedMetadata > 291.229 org.apache.hudi.table.TestHoodieMergeOnReadTable > 245.922 org.apache.hudi.functional.TestStructuredStreaming > 234.737 org.apache.hudi.index.TestHoodieIndex > 224.893 org.apache.hudi.functional.TestMORDataSource > 217.354 > org.apache.hudi.spark3.internal.TestHoodieBulkInsertDataInternalWriter > 210.717 org.apache.hudi.common.functional.TestHoodieLogFormat > 204.989 org.apache.hudi.table.TestCleaner > 198.623 org.apache.hudi.client.TestBootstrap > 170.029 org.apache.hudi.hive.TestHiveSyncTool > 135.295 org.apache.hudi.table.action.compact.TestInlineCompaction > 116.381 org.apache.hudi.table.action.commit.TestCopyOnWriteActionExecutor > 113.559 org.apache.hudi.functional.TestCOWDataSource > 111.297 org.apache.hudi.table.upgrade.TestUpgradeDowngrade > > Thanks > Vinoth > -- Regards, -Sivabalan
