[ https://issues.apache.org/jira/browse/HUDI-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymond Xu reopened HUDI-3043: ------------------------------ > Test failure with > TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWriters > ----------------------------------------------------------------------------------------------------- > > Key: HUDI-3043 > URL: https://issues.apache.org/jira/browse/HUDI-3043 > Project: Apache Hudi > Issue Type: Task > Affects Versions: 0.11.0 > Reporter: sivabalan narayanan > Assignee: sivabalan narayanan > Priority: Major > Labels: pull-request-available > Fix For: 0.11.0, 0.10.1 > > Attachments: Screen Shot 2021-12-17 at 3.07.39 PM.png > > > CI has been very flaky with this test. > reference: > [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/4431/logs/35] > > Test fails after a timeout of 360 secs. This is what the test is doing. > Instantiate two concurrent deltastreamer jobs. But are expected to conflict > and one of them should fail. > > {code:java} > 2021-12-15T12:31:08.1443043Z 415268 [pool-46-thread-1] WARN > org.apache.hudi.common.table.HoodieTableConfig - Run `table recover-configs` > if config update/delete failed midway. Falling back to backed up configs. > 2021-12-15T12:31:08.1479662Z 415272 [pool-46-thread-1] WARN > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error : > 2021-12-15T12:31:08.1480755Z org.apache.hudi.exception.HoodieIOException: > Could not load Hoodie properties from > /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties > 2021-12-15T12:31:08.1481566Z at > org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190) > 2021-12-15T12:31:08.1482274Z at > org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114) > 2021-12-15T12:31:08.1483350Z at > org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73) > 2021-12-15T12:31:08.1484084Z at > org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614) > 2021-12-15T12:31:08.1484912Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315) > 2021-12-15T12:31:08.1485842Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312) > 2021-12-15T12:31:08.1486735Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340) > 2021-12-15T12:31:08.1487473Z at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > 2021-12-15T12:31:08.1488117Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2021-12-15T12:31:08.1488795Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2021-12-15T12:31:08.1489389Z at java.lang.Thread.run(Thread.java:748) > 2021-12-15T12:31:08.1490080Z Caused by: java.io.FileNotFoundException: File > file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup > does not exist > 2021-12-15T12:31:08.1490870Z at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611) > 2021-12-15T12:31:08.1491603Z at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824) > 2021-12-15T12:31:08.1492301Z at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601) > 2021-12-15T12:31:08.1492988Z at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) > 2021-12-15T12:31:08.1493700Z at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142) > 2021-12-15T12:31:08.1494397Z at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346) > 2021-12-15T12:31:08.1495022Z at > org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769) > 2021-12-15T12:31:08.1495676Z at > org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459) > 2021-12-15T12:31:08.1496385Z at > org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212) > 2021-12-15T12:31:08.1497081Z at > org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180) > 2021-12-15T12:31:08.1497619Z ... 10 more > 2021-12-15T12:31:09.0533739Z 416177 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit1940536249034667747/source/.hoodie/metadata > 2021-12-15T12:31:09.0821375Z 416206 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit1940536249034667747/source/.hoodie/metadata > 2021-12-15T12:31:09.4001348Z 06:50 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:09.4012877Z 06:50 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:09.4794927Z 06:50 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:11.4361520Z 06:52 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:12.3883215Z 419512 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit6695759085666687592/source/.hoodie/metadata > 2021-12-15T12:31:12.4075450Z 419531 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit6695759085666687592/source/.hoodie/metadata > 2021-12-15T12:31:12.7802513Z 06:54 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:12.7847857Z 06:54 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:12.8628212Z 06:54 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:13.1489569Z 420273 [pool-46-thread-1] WARN > org.apache.hudi.common.table.HoodieTableConfig - Run `table recover-configs` > if config update/delete failed midway. Falling back to backed up configs. > 2021-12-15T12:31:13.1502918Z 420273 [pool-46-thread-1] WARN > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error : > 2021-12-15T12:31:13.1503992Z org.apache.hudi.exception.HoodieIOException: > Could not load Hoodie properties from > /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties > 2021-12-15T12:31:13.1504809Z at > org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190) > 2021-12-15T12:31:13.1505526Z at > org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114) > 2021-12-15T12:31:13.1506253Z at > org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73) > 2021-12-15T12:31:13.1506983Z at > org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614) > 2021-12-15T12:31:13.1507927Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315) > 2021-12-15T12:31:13.1508853Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312) > 2021-12-15T12:31:13.1509879Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340) > 2021-12-15T12:31:13.1510645Z at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > 2021-12-15T12:31:13.1511272Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2021-12-15T12:31:13.1511948Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2021-12-15T12:31:13.1512551Z at java.lang.Thread.run(Thread.java:748) > 2021-12-15T12:31:13.1513607Z Caused by: java.io.FileNotFoundException: File > file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup > does not exist > 2021-12-15T12:31:13.1514412Z at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611) > 2021-12-15T12:31:13.1515141Z at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824) > 2021-12-15T12:31:13.1515860Z at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601) > 2021-12-15T12:31:13.1516555Z at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) > 2021-12-15T12:31:13.1518333Z at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142) > 2021-12-15T12:31:13.1519089Z at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346) > 2021-12-15T12:31:13.1519729Z at > org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769) > 2021-12-15T12:31:13.1520387Z at > org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459) > 2021-12-15T12:31:13.1521094Z at > org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212) > 2021-12-15T12:31:13.1521791Z at > org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180) > 2021-12-15T12:31:13.1522342Z ... 10 more > 2021-12-15T12:31:14.7650788Z 06:56 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:15.5436369Z 422667 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit4191731663987779053/source/.hoodie/metadata > 2021-12-15T12:31:15.5616963Z 422686 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit4191731663987779053/source/.hoodie/metadata > 2021-12-15T12:31:15.8443523Z 06:57 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:15.8454420Z 06:57 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:15.9309839Z 06:57 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:17.8366064Z 06:59 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:18.1515156Z 425275 [pool-46-thread-1] WARN > org.apache.hudi.common.table.HoodieTableConfig - Run `table recover-configs` > if config update/delete failed midway. Falling back to backed up configs. > 2021-12-15T12:31:18.1516556Z 425275 [pool-46-thread-1] WARN > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error : > 2021-12-15T12:31:18.1517464Z org.apache.hudi.exception.HoodieIOException: > Could not load Hoodie properties from > /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties > 2021-12-15T12:31:18.1518265Z at > org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190) > 2021-12-15T12:31:18.1519344Z at > org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114) > 2021-12-15T12:31:18.1519949Z at > org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73) > 2021-12-15T12:31:18.1520520Z at > org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614) > 2021-12-15T12:31:18.1521158Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315) > 2021-12-15T12:31:18.1521875Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312) > 2021-12-15T12:31:18.1522917Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340) > 2021-12-15T12:31:18.1523479Z at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > 2021-12-15T12:31:18.1523916Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2021-12-15T12:31:18.1524404Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2021-12-15T12:31:18.1524820Z at java.lang.Thread.run(Thread.java:748) > 2021-12-15T12:31:18.1525313Z Caused by: java.io.FileNotFoundException: File > file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup > does not exist > 2021-12-15T12:31:18.1525956Z at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611) > 2021-12-15T12:31:18.1526494Z at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824) > 2021-12-15T12:31:18.1527013Z at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601) > 2021-12-15T12:31:18.1527495Z at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) > 2021-12-15T12:31:18.1528014Z at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142) > 2021-12-15T12:31:18.1528523Z at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346) > 2021-12-15T12:31:18.1528952Z at > org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769) > 2021-12-15T12:31:18.1529418Z at > org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459) > 2021-12-15T12:31:18.1530054Z at > org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212) > 2021-12-15T12:31:18.1530560Z at > org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180) > 2021-12-15T12:31:18.1530912Z ... 10 more > 2021-12-15T12:31:18.6080652Z 425732 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit8182236215041118309/source/.hoodie/metadata > 2021-12-15T12:31:18.6248321Z 425749 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit8182236215041118309/source/.hoodie/metadata > 2021-12-15T12:31:18.9049323Z 07:00 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:18.9086840Z 07:00 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:18.9773610Z 07:00 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:20.8238902Z 07:02 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:21.8356036Z 428959 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit7367394654999483124/source/.hoodie/metadata > 2021-12-15T12:31:21.8519931Z 428976 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit7367394654999483124/source/.hoodie/metadata > 2021-12-15T12:31:22.1387636Z 07:03 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:22.1442159Z 07:03 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:22.1973498Z 07:03 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:23.1526405Z 430276 [pool-46-thread-1] WARN > org.apache.hudi.common.table.HoodieTableConfig - Run `table recover-configs` > if config update/delete failed midway. Falling back to backed up configs. > 2021-12-15T12:31:23.1527490Z 430276 [pool-46-thread-1] WARN > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error : > 2021-12-15T12:31:23.1528146Z org.apache.hudi.exception.HoodieIOException: > Could not load Hoodie properties from > /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties > 2021-12-15T12:31:23.1528881Z at > org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190) > 2021-12-15T12:31:23.1529399Z at > org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114) > 2021-12-15T12:31:23.1529934Z at > org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73) > 2021-12-15T12:31:23.1530477Z at > org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614) > 2021-12-15T12:31:23.1531100Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315) > 2021-12-15T12:31:23.1531842Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312) > 2021-12-15T12:31:23.1532547Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340) > 2021-12-15T12:31:23.1533092Z at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > 2021-12-15T12:31:23.1533647Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2021-12-15T12:31:23.1534137Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2021-12-15T12:31:23.1534554Z at java.lang.Thread.run(Thread.java:748) > 2021-12-15T12:31:23.1535044Z Caused by: java.io.FileNotFoundException: File > file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup > does not exist > 2021-12-15T12:31:23.1535642Z at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611) > 2021-12-15T12:31:23.1536186Z at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824) > 2021-12-15T12:31:23.1536689Z at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601) > 2021-12-15T12:31:23.1537183Z at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) > 2021-12-15T12:31:23.1537702Z at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142) > 2021-12-15T12:31:23.1538216Z at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346) > 2021-12-15T12:31:23.1538644Z at > org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769) > 2021-12-15T12:31:23.1539108Z at > org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459) > 2021-12-15T12:31:23.1539625Z at > org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212) > 2021-12-15T12:31:23.1540118Z at > org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180) > 2021-12-15T12:31:23.1540484Z ... 10 more > 2021-12-15T12:31:24.0393154Z 07:05 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:24.2768054Z 431401 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata > 2021-12-15T12:31:24.3011474Z 431423 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata > 2021-12-15T12:31:24.5931674Z 07:05 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:24.5982776Z 07:05 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:24.6715314Z 07:06 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:26.4723566Z 07:07 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:26.6296934Z 433752 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata > 2021-12-15T12:31:26.6990277Z 433823 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit8519882895692188385/source/.hoodie/metadata > 2021-12-15T12:31:26.7154263Z 433839 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit8519882895692188385/source/.hoodie/metadata > 2021-12-15T12:31:27.0226015Z 07:08 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:27.0285268Z 07:08 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:27.0977866Z 07:08 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:28.1534844Z 435277 [pool-46-thread-1] WARN > org.apache.hudi.common.table.HoodieTableConfig - Run `table recover-configs` > if config update/delete failed midway. Falling back to backed up configs. > 2021-12-15T12:31:28.1536264Z 435277 [pool-46-thread-1] WARN > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error : > 2021-12-15T12:31:28.1537195Z org.apache.hudi.exception.HoodieIOException: > Could not load Hoodie properties from > /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties > 2021-12-15T12:31:28.1537984Z at > org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190) > 2021-12-15T12:31:28.1539028Z at > org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114) > 2021-12-15T12:31:28.1539640Z at > org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73) > 2021-12-15T12:31:28.1540183Z at > org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614) > 2021-12-15T12:31:28.1540885Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315) > 2021-12-15T12:31:28.1541621Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312) > 2021-12-15T12:31:28.1542337Z at > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340) > 2021-12-15T12:31:28.1542884Z at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > 2021-12-15T12:31:28.1543318Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2021-12-15T12:31:28.1543805Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2021-12-15T12:31:28.1544221Z at java.lang.Thread.run(Thread.java:748) > 2021-12-15T12:31:28.1544706Z Caused by: java.io.FileNotFoundException: File > file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup > does not exist > 2021-12-15T12:31:28.1545306Z at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611) > 2021-12-15T12:31:28.1545840Z at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824) > 2021-12-15T12:31:28.1546345Z at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601) > 2021-12-15T12:31:28.1547105Z at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) > 2021-12-15T12:31:28.1547635Z at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142) > 2021-12-15T12:31:28.1548151Z at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346) > 2021-12-15T12:31:28.1548578Z at > org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769) > 2021-12-15T12:31:28.1549239Z at > org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459) > 2021-12-15T12:31:28.1549777Z at > org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212) > 2021-12-15T12:31:28.1550273Z at > org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180) > 2021-12-15T12:31:28.1550655Z ... 10 more > 2021-12-15T12:31:28.8646121Z 07:10 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:29.0315217Z 436154 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit6507215968345229941/source/.hoodie/metadata > 2021-12-15T12:31:29.0474647Z 436171 [main] WARN > org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not > found at path file:/tmp/junit6507215968345229941/source/.hoodie/metadata > 2021-12-15T12:31:29.9039470Z 07:10 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:29.9040844Z 07:10 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:29.9041727Z 07:10 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:31.2335559Z 07:12 WARN: Timeline-server-based markers are > configured as the marker type but embedded timeline server is not enabled. > Falling back to direct markers. > 2021-12-15T12:31:32.1873178Z [ERROR] Tests run: 26, Failures: 0, Errors: 1, > Skipped: 9, Time elapsed: 439.775 s <<< FAILURE! - in JUnit Vintage > 2021-12-15T12:31:32.1874589Z [ERROR] HoodieTableType).[2] > MERGE_ON_READ(testUpsertsContinuousModeWithMultipleWriters Time elapsed: > 360.847 s <<< ERROR! > 2021-12-15T12:31:32.1875409Z java.util.concurrent.ExecutionException: > java.lang.RuntimeException: java.util.concurrent.TimeoutException > 2021-12-15T12:31:32.1876498Z Caused by: java.lang.RuntimeException: > java.util.concurrent.TimeoutException > 2021-12-15T12:31:32.1877024Z Caused by: java.util.concurrent.TimeoutException > 2021-12-15T12:31:32.1877203Z > 2021-12-15T12:31:32.5675718Z [INFO] > 2021-12-15T12:31:32.5710112Z [INFO] Results: > 2021-12-15T12:31:32.5710772Z [INFO] > 2021-12-15T12:31:32.5711020Z [ERROR] Errors: > 2021-12-15T12:31:32.5711359Z [ERROR] java.lang.RuntimeException: > java.util.concurrent.TimeoutException > 2021-12-15T12:31:32.5711695Z [INFO] > 2021-12-15T12:31:32.5712044Z [ERROR] Tests run: 26, Failures: 0, Errors: 1, > Skipped: 9 {code} -- This message was sent by Atlassian Jira (v8.20.1#820001)