[ https://issues.apache.org/jira/browse/HUDI-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan updated HUDI-6445: -------------------------------------- Description: CI has been unstable for the past few weeks. we need to triage them and fix it. UT-spark datasource module times out after 3 hours. [https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=17956&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc] * Looks like top 10 tests were taking 30 to 40 secs and now its taking 40 to 50 secs or more and hence reaching 3 hours {code:java} 2023-06-20T05:03:58.6566739Z 52.124 org.apache.hudi.functional.TestIncrementalReadWithFullTableScan testFailEarlyForIncrViewQue ryForNonExistingFiles{HoodieTableType}[2] 2023-06-20T05:03:58.6567324Z 49.446 org.apache.hudi.functional.TestIncrementalReadWithFullTableScan testFailEarlyForIncrViewQue ryForNonExistingFiles{HoodieTableType}[1] 2023-06-20T05:03:58.6568005Z 48.659 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testMORDataSourceWrite{HoodieCDCSupple mentalLoggingMode}[1] 2023-06-20T05:03:58.6568471Z 47.799 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testMORDataSourceWrite{HoodieCDCSupple mentalLoggingMode}[3] 2023-06-20T05:03:58.6569093Z 47.586 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testMORDataSourceWrite{HoodieCDCSupple mentalLoggingMode}[2] 2023-06-20T05:03:58.6569503Z 41.208 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, String}[2] 2023-06-20T05:03:58.6570090Z 41.034 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, String}[4] 2023-06-20T05:03:58.6570501Z 40.225 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, String}[3] 2023-06-20T05:03:58.6571231Z 39.853 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testCOWDataSourceWrite{HoodieCDCSupple mentalLoggingMode}[1] 2023-06-20T05:03:58.6574224Z 39.357 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, String}[1] 2023-06-20T05:03:58.6575261Z 38.995 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testCOWDataSourceWrite{HoodieCDCSupple mentalLoggingMode}[3] 2023-06-20T05:03:58.6575765Z 38.846 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testCOWDataSourceWrite{HoodieCDCSupple mentalLoggingMode}[2] 2023-06-20T05:03:58.6576470Z 35.404 org.apache.hudi.functional.TestMORDataSourceWithBucketIndex testCountWithBucketIndex {code} TestHoodieDeltaStreamer.testUpsertsMORContinuousMode and testAsyncClusteringServiceWithCompaction [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18111/logs/19] TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWritersForConflicts [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18080/logs/21] TestWriteMergeOnRead.testUpsert [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/17993/logs/35] TestWriteMergeOnReadWithCompact.testUpsert TestWriteCopyOnWrite.testSubtaskFails [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18110/logs/30] was: CI has been unstable for the past few weeks. we need to triage them and fix it. UT-spark datasource module times out after 3 hours. [https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=17956&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc] TestHoodieDeltaStreamer.testUpsertsMORContinuousMode and testAsyncClusteringServiceWithCompaction [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18111/logs/19] TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWritersForConflicts [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18080/logs/21] TestWriteMergeOnRead.testUpsert [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/17993/logs/35] TestWriteMergeOnReadWithCompact.testUpsert TestWriteCopyOnWrite.testSubtaskFails [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18110/logs/30] > Fix CI stability Jun 26, 2023 > ----------------------------- > > Key: HUDI-6445 > URL: https://issues.apache.org/jira/browse/HUDI-6445 > Project: Apache Hudi > Issue Type: Test > Components: tests-ci > Reporter: sivabalan narayanan > Priority: Major > > CI has been unstable for the past few weeks. we need to triage them and fix > it. > > > UT-spark datasource module times out after 3 hours. > [https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=17956&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc] > * Looks like top 10 tests were taking 30 to 40 secs and now its taking 40 to > 50 secs or more and hence reaching 3 hours > {code:java} > 2023-06-20T05:03:58.6566739Z 52.124 > org.apache.hudi.functional.TestIncrementalReadWithFullTableScan > testFailEarlyForIncrViewQue > ryForNonExistingFiles{HoodieTableType}[2] > 2023-06-20T05:03:58.6567324Z 49.446 > org.apache.hudi.functional.TestIncrementalReadWithFullTableScan > testFailEarlyForIncrViewQue > ryForNonExistingFiles{HoodieTableType}[1] > 2023-06-20T05:03:58.6568005Z 48.659 > org.apache.hudi.functional.cdc.TestCDCDataFrameSuite > testMORDataSourceWrite{HoodieCDCSupple > mentalLoggingMode}[1] > 2023-06-20T05:03:58.6568471Z 47.799 > org.apache.hudi.functional.cdc.TestCDCDataFrameSuite > testMORDataSourceWrite{HoodieCDCSupple > mentalLoggingMode}[3] > 2023-06-20T05:03:58.6569093Z 47.586 > org.apache.hudi.functional.cdc.TestCDCDataFrameSuite > testMORDataSourceWrite{HoodieCDCSupple > mentalLoggingMode}[2] > 2023-06-20T05:03:58.6569503Z 41.208 > org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, > HoodieRecordType, > String}[2] > 2023-06-20T05:03:58.6570090Z 41.034 > org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, > HoodieRecordType, > String}[4] > 2023-06-20T05:03:58.6570501Z 40.225 > org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, > HoodieRecordType, > String}[3] > 2023-06-20T05:03:58.6571231Z 39.853 > org.apache.hudi.functional.cdc.TestCDCDataFrameSuite > testCOWDataSourceWrite{HoodieCDCSupple > mentalLoggingMode}[1] > 2023-06-20T05:03:58.6574224Z 39.357 > org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, > HoodieRecordType, > String}[1] > 2023-06-20T05:03:58.6575261Z 38.995 > org.apache.hudi.functional.cdc.TestCDCDataFrameSuite > testCOWDataSourceWrite{HoodieCDCSupple > mentalLoggingMode}[3] > 2023-06-20T05:03:58.6575765Z 38.846 > org.apache.hudi.functional.cdc.TestCDCDataFrameSuite > testCOWDataSourceWrite{HoodieCDCSupple > mentalLoggingMode}[2] > 2023-06-20T05:03:58.6576470Z 35.404 > org.apache.hudi.functional.TestMORDataSourceWithBucketIndex > testCountWithBucketIndex {code} > > TestHoodieDeltaStreamer.testUpsertsMORContinuousMode > and testAsyncClusteringServiceWithCompaction > [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18111/logs/19] > > TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWritersForConflicts > [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18080/logs/21] > > TestWriteMergeOnRead.testUpsert > [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/17993/logs/35] > > TestWriteMergeOnReadWithCompact.testUpsert > TestWriteCopyOnWrite.testSubtaskFails > [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18110/logs/30] > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)