[ 
https://issues.apache.org/jira/browse/HUDI-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raymond Xu reopened HUDI-3043:
------------------------------

> Test failure with 
> TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWriters
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HUDI-3043
>                 URL: https://issues.apache.org/jira/browse/HUDI-3043
>             Project: Apache Hudi
>          Issue Type: Task
>    Affects Versions: 0.11.0
>            Reporter: sivabalan narayanan
>            Assignee: sivabalan narayanan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 0.11.0, 0.10.1
>
>         Attachments: Screen Shot 2021-12-17 at 3.07.39 PM.png
>
>
> CI has been very flaky with this test. 
> reference: 
> [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/4431/logs/35]
>  
> Test fails after a timeout of 360 secs. This is what the test is doing. 
> Instantiate two concurrent deltastreamer jobs. But are expected to conflict 
> and one of them should fail.
>  
> {code:java}
> 2021-12-15T12:31:08.1443043Z 415268 [pool-46-thread-1] WARN  
> org.apache.hudi.common.table.HoodieTableConfig  - Run `table recover-configs` 
> if config update/delete failed midway. Falling back to backed up configs.
> 2021-12-15T12:31:08.1479662Z 415272 [pool-46-thread-1] WARN  
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer  - Got error :
> 2021-12-15T12:31:08.1480755Z org.apache.hudi.exception.HoodieIOException: 
> Could not load Hoodie properties from 
> /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
> 2021-12-15T12:31:08.1481566Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
> 2021-12-15T12:31:08.1482274Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
> 2021-12-15T12:31:08.1483350Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
> 2021-12-15T12:31:08.1484084Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
> 2021-12-15T12:31:08.1484912Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
> 2021-12-15T12:31:08.1485842Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
> 2021-12-15T12:31:08.1486735Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
> 2021-12-15T12:31:08.1487473Z  at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-12-15T12:31:08.1488117Z  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-12-15T12:31:08.1488795Z  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-12-15T12:31:08.1489389Z  at java.lang.Thread.run(Thread.java:748)
> 2021-12-15T12:31:08.1490080Z Caused by: java.io.FileNotFoundException: File 
> file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup
>  does not exist
> 2021-12-15T12:31:08.1490870Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> 2021-12-15T12:31:08.1491603Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> 2021-12-15T12:31:08.1492301Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> 2021-12-15T12:31:08.1492988Z  at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> 2021-12-15T12:31:08.1493700Z  at 
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> 2021-12-15T12:31:08.1494397Z  at 
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
> 2021-12-15T12:31:08.1495022Z  at 
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> 2021-12-15T12:31:08.1495676Z  at 
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
> 2021-12-15T12:31:08.1496385Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
> 2021-12-15T12:31:08.1497081Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
> 2021-12-15T12:31:08.1497619Z  ... 10 more
> 2021-12-15T12:31:09.0533739Z 416177 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit1940536249034667747/source/.hoodie/metadata
> 2021-12-15T12:31:09.0821375Z 416206 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit1940536249034667747/source/.hoodie/metadata
> 2021-12-15T12:31:09.4001348Z 06:50  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:09.4012877Z 06:50  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:09.4794927Z 06:50  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:11.4361520Z 06:52  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:12.3883215Z 419512 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit6695759085666687592/source/.hoodie/metadata
> 2021-12-15T12:31:12.4075450Z 419531 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit6695759085666687592/source/.hoodie/metadata
> 2021-12-15T12:31:12.7802513Z 06:54  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:12.7847857Z 06:54  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:12.8628212Z 06:54  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:13.1489569Z 420273 [pool-46-thread-1] WARN  
> org.apache.hudi.common.table.HoodieTableConfig  - Run `table recover-configs` 
> if config update/delete failed midway. Falling back to backed up configs.
> 2021-12-15T12:31:13.1502918Z 420273 [pool-46-thread-1] WARN  
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer  - Got error :
> 2021-12-15T12:31:13.1503992Z org.apache.hudi.exception.HoodieIOException: 
> Could not load Hoodie properties from 
> /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
> 2021-12-15T12:31:13.1504809Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
> 2021-12-15T12:31:13.1505526Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
> 2021-12-15T12:31:13.1506253Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
> 2021-12-15T12:31:13.1506983Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
> 2021-12-15T12:31:13.1507927Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
> 2021-12-15T12:31:13.1508853Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
> 2021-12-15T12:31:13.1509879Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
> 2021-12-15T12:31:13.1510645Z  at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-12-15T12:31:13.1511272Z  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-12-15T12:31:13.1511948Z  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-12-15T12:31:13.1512551Z  at java.lang.Thread.run(Thread.java:748)
> 2021-12-15T12:31:13.1513607Z Caused by: java.io.FileNotFoundException: File 
> file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup
>  does not exist
> 2021-12-15T12:31:13.1514412Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> 2021-12-15T12:31:13.1515141Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> 2021-12-15T12:31:13.1515860Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> 2021-12-15T12:31:13.1516555Z  at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> 2021-12-15T12:31:13.1518333Z  at 
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> 2021-12-15T12:31:13.1519089Z  at 
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
> 2021-12-15T12:31:13.1519729Z  at 
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> 2021-12-15T12:31:13.1520387Z  at 
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
> 2021-12-15T12:31:13.1521094Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
> 2021-12-15T12:31:13.1521791Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
> 2021-12-15T12:31:13.1522342Z  ... 10 more
> 2021-12-15T12:31:14.7650788Z 06:56  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:15.5436369Z 422667 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit4191731663987779053/source/.hoodie/metadata
> 2021-12-15T12:31:15.5616963Z 422686 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit4191731663987779053/source/.hoodie/metadata
> 2021-12-15T12:31:15.8443523Z 06:57  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:15.8454420Z 06:57  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:15.9309839Z 06:57  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:17.8366064Z 06:59  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:18.1515156Z 425275 [pool-46-thread-1] WARN  
> org.apache.hudi.common.table.HoodieTableConfig  - Run `table recover-configs` 
> if config update/delete failed midway. Falling back to backed up configs.
> 2021-12-15T12:31:18.1516556Z 425275 [pool-46-thread-1] WARN  
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer  - Got error :
> 2021-12-15T12:31:18.1517464Z org.apache.hudi.exception.HoodieIOException: 
> Could not load Hoodie properties from 
> /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
> 2021-12-15T12:31:18.1518265Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
> 2021-12-15T12:31:18.1519344Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
> 2021-12-15T12:31:18.1519949Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
> 2021-12-15T12:31:18.1520520Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
> 2021-12-15T12:31:18.1521158Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
> 2021-12-15T12:31:18.1521875Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
> 2021-12-15T12:31:18.1522917Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
> 2021-12-15T12:31:18.1523479Z  at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-12-15T12:31:18.1523916Z  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-12-15T12:31:18.1524404Z  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-12-15T12:31:18.1524820Z  at java.lang.Thread.run(Thread.java:748)
> 2021-12-15T12:31:18.1525313Z Caused by: java.io.FileNotFoundException: File 
> file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup
>  does not exist
> 2021-12-15T12:31:18.1525956Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> 2021-12-15T12:31:18.1526494Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> 2021-12-15T12:31:18.1527013Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> 2021-12-15T12:31:18.1527495Z  at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> 2021-12-15T12:31:18.1528014Z  at 
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> 2021-12-15T12:31:18.1528523Z  at 
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
> 2021-12-15T12:31:18.1528952Z  at 
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> 2021-12-15T12:31:18.1529418Z  at 
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
> 2021-12-15T12:31:18.1530054Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
> 2021-12-15T12:31:18.1530560Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
> 2021-12-15T12:31:18.1530912Z  ... 10 more
> 2021-12-15T12:31:18.6080652Z 425732 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit8182236215041118309/source/.hoodie/metadata
> 2021-12-15T12:31:18.6248321Z 425749 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit8182236215041118309/source/.hoodie/metadata
> 2021-12-15T12:31:18.9049323Z 07:00  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:18.9086840Z 07:00  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:18.9773610Z 07:00  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:20.8238902Z 07:02  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:21.8356036Z 428959 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit7367394654999483124/source/.hoodie/metadata
> 2021-12-15T12:31:21.8519931Z 428976 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit7367394654999483124/source/.hoodie/metadata
> 2021-12-15T12:31:22.1387636Z 07:03  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:22.1442159Z 07:03  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:22.1973498Z 07:03  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:23.1526405Z 430276 [pool-46-thread-1] WARN  
> org.apache.hudi.common.table.HoodieTableConfig  - Run `table recover-configs` 
> if config update/delete failed midway. Falling back to backed up configs.
> 2021-12-15T12:31:23.1527490Z 430276 [pool-46-thread-1] WARN  
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer  - Got error :
> 2021-12-15T12:31:23.1528146Z org.apache.hudi.exception.HoodieIOException: 
> Could not load Hoodie properties from 
> /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
> 2021-12-15T12:31:23.1528881Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
> 2021-12-15T12:31:23.1529399Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
> 2021-12-15T12:31:23.1529934Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
> 2021-12-15T12:31:23.1530477Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
> 2021-12-15T12:31:23.1531100Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
> 2021-12-15T12:31:23.1531842Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
> 2021-12-15T12:31:23.1532547Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
> 2021-12-15T12:31:23.1533092Z  at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-12-15T12:31:23.1533647Z  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-12-15T12:31:23.1534137Z  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-12-15T12:31:23.1534554Z  at java.lang.Thread.run(Thread.java:748)
> 2021-12-15T12:31:23.1535044Z Caused by: java.io.FileNotFoundException: File 
> file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup
>  does not exist
> 2021-12-15T12:31:23.1535642Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> 2021-12-15T12:31:23.1536186Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> 2021-12-15T12:31:23.1536689Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> 2021-12-15T12:31:23.1537183Z  at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> 2021-12-15T12:31:23.1537702Z  at 
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> 2021-12-15T12:31:23.1538216Z  at 
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
> 2021-12-15T12:31:23.1538644Z  at 
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> 2021-12-15T12:31:23.1539108Z  at 
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
> 2021-12-15T12:31:23.1539625Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
> 2021-12-15T12:31:23.1540118Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
> 2021-12-15T12:31:23.1540484Z  ... 10 more
> 2021-12-15T12:31:24.0393154Z 07:05  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:24.2768054Z 431401 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata
> 2021-12-15T12:31:24.3011474Z 431423 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata
> 2021-12-15T12:31:24.5931674Z 07:05  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:24.5982776Z 07:05  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:24.6715314Z 07:06  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:26.4723566Z 07:07  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:26.6296934Z 433752 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata
> 2021-12-15T12:31:26.6990277Z 433823 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit8519882895692188385/source/.hoodie/metadata
> 2021-12-15T12:31:26.7154263Z 433839 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit8519882895692188385/source/.hoodie/metadata
> 2021-12-15T12:31:27.0226015Z 07:08  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:27.0285268Z 07:08  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:27.0977866Z 07:08  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:28.1534844Z 435277 [pool-46-thread-1] WARN  
> org.apache.hudi.common.table.HoodieTableConfig  - Run `table recover-configs` 
> if config update/delete failed midway. Falling back to backed up configs.
> 2021-12-15T12:31:28.1536264Z 435277 [pool-46-thread-1] WARN  
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer  - Got error :
> 2021-12-15T12:31:28.1537195Z org.apache.hudi.exception.HoodieIOException: 
> Could not load Hoodie properties from 
> /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
> 2021-12-15T12:31:28.1537984Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
> 2021-12-15T12:31:28.1539028Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
> 2021-12-15T12:31:28.1539640Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
> 2021-12-15T12:31:28.1540183Z  at 
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
> 2021-12-15T12:31:28.1540885Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
> 2021-12-15T12:31:28.1541621Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
> 2021-12-15T12:31:28.1542337Z  at 
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
> 2021-12-15T12:31:28.1542884Z  at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-12-15T12:31:28.1543318Z  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-12-15T12:31:28.1543805Z  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-12-15T12:31:28.1544221Z  at java.lang.Thread.run(Thread.java:748)
> 2021-12-15T12:31:28.1544706Z Caused by: java.io.FileNotFoundException: File 
> file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup
>  does not exist
> 2021-12-15T12:31:28.1545306Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> 2021-12-15T12:31:28.1545840Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> 2021-12-15T12:31:28.1546345Z  at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> 2021-12-15T12:31:28.1547105Z  at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> 2021-12-15T12:31:28.1547635Z  at 
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> 2021-12-15T12:31:28.1548151Z  at 
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
> 2021-12-15T12:31:28.1548578Z  at 
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> 2021-12-15T12:31:28.1549239Z  at 
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
> 2021-12-15T12:31:28.1549777Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
> 2021-12-15T12:31:28.1550273Z  at 
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
> 2021-12-15T12:31:28.1550655Z  ... 10 more
> 2021-12-15T12:31:28.8646121Z 07:10  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:29.0315217Z 436154 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit6507215968345229941/source/.hoodie/metadata
> 2021-12-15T12:31:29.0474647Z 436171 [main] WARN  
> org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not 
> found at path file:/tmp/junit6507215968345229941/source/.hoodie/metadata
> 2021-12-15T12:31:29.9039470Z 07:10  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:29.9040844Z 07:10  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:29.9041727Z 07:10  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:31.2335559Z 07:12  WARN: Timeline-server-based markers are 
> configured as the marker type but embedded timeline server is not enabled.  
> Falling back to direct markers.
> 2021-12-15T12:31:32.1873178Z [ERROR] Tests run: 26, Failures: 0, Errors: 1, 
> Skipped: 9, Time elapsed: 439.775 s <<< FAILURE! - in JUnit Vintage
> 2021-12-15T12:31:32.1874589Z [ERROR] HoodieTableType).[2] 
> MERGE_ON_READ(testUpsertsContinuousModeWithMultipleWriters  Time elapsed: 
> 360.847 s  <<< ERROR!
> 2021-12-15T12:31:32.1875409Z java.util.concurrent.ExecutionException: 
> java.lang.RuntimeException: java.util.concurrent.TimeoutException
> 2021-12-15T12:31:32.1876498Z Caused by: java.lang.RuntimeException: 
> java.util.concurrent.TimeoutException
> 2021-12-15T12:31:32.1877024Z Caused by: java.util.concurrent.TimeoutException
> 2021-12-15T12:31:32.1877203Z 
> 2021-12-15T12:31:32.5675718Z [INFO] 
> 2021-12-15T12:31:32.5710112Z [INFO] Results:
> 2021-12-15T12:31:32.5710772Z [INFO] 
> 2021-12-15T12:31:32.5711020Z [ERROR] Errors: 
> 2021-12-15T12:31:32.5711359Z [ERROR]   java.lang.RuntimeException: 
> java.util.concurrent.TimeoutException
> 2021-12-15T12:31:32.5711695Z [INFO] 
> 2021-12-15T12:31:32.5712044Z [ERROR] Tests run: 26, Failures: 0, Errors: 1, 
> Skipped: 9 {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Reply via email to