alberttwong opened a new issue, #460: URL: https://github.com/apache/incubator-xtable/issues/460
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues. ### Please describe the bug 🐞 I wrote 5 rows into delta on s3 and then used xtable to covert the data to iceberg and hudi. Next I used run_sync_tool.sh to register the hudi files in HMS and then query it into Trino. No data in Trino. xtable looks good ``` 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[00000000000000011__deltacommit__COMPLETED__20240604232709963]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadataWriter:451 - Initializing COLUMN_STATS index in metadata table took 145 in ms 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieTableMetadataUtil:395 - Updating at 20240604232400230 from Commit/null. #partitions_updated=6, #files_added=6 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieTableMetadataUtil:942 - Loading latest file slices for metadata table partition column_stats 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (column_stats) 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieTableMetadataUtil:942 - Loading latest file slices for metadata table partition files 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (files) 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[00000000000000011__deltacommit__COMPLETED__20240604232709963]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadataWriter:1103 - New commit at 20240604232400230 being applied to MDT. 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[00000000000000011__deltacommit__COMPLETED__20240604232709963]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.CleanerUtils:221 - Cleaned failed attempts if any 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[00000000000000011__deltacommit__COMPLETED__20240604232709963]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View 2024-06-04 23:27:10 INFO org.apache.hudi.client.BaseHoodieWriteClient:945 - Generate a new instant time: 20240604232400230 action: deltacommit 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:202 - Creating a new instant [==>20240604232400230__deltacommit__REQUESTED] 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:615 - Checking for file exists ?s3a://warehouse/people/.hoodie/metadata/.hoodie/20240604232400230.deltacommit.requested 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:623 - Create new file for toInstant ?s3a://warehouse/people/.hoodie/metadata/.hoodie/20240604232400230.deltacommit.inflight 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[==>20240604232400230__deltacommit__INFLIGHT__20240604232710097]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View 2024-06-04 23:27:10 INFO org.apache.hudi.async.AsyncCleanerService:63 - The HoodieWriteClient is not configured to auto & async clean. Async clean service will not start. 2024-06-04 23:27:10 INFO org.apache.hudi.async.AsyncArchiveService:61 - The HoodieWriteClient is not configured to auto & async archive. Async archive service will not start. 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.deltacommit.JavaUpsertPreppedDeltaCommitActionExecutor:74 - Total update fileIDs 3, total inserts 0 for commit 20240604232400230 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:165 - Creating InMemory based view for basePath s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (files) # WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf # WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed.] 2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:197 - Creating Marker Path=s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/files/files-0000-0_0-0-0_00000000000000010.hfile.marker.APPEND 2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:202 - [direct] Created marker file s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/files/files-0000-0_0-0-0_00000000000000010.hfile.marker.APPEND in 17 ms 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:215 - Building HoodieLogFormat Writer 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:270 - HoodieLogFile on path s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatWriter:122 - Append not supported.. Rolling over to HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.io.HoodieAppendHandle:438 - AppendHandle for partitionPath files filePath files/.files-0000-0_00000000000000010.log.2_0-0-0, took 301 ms. 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (column_stats) 2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:197 - Creating Marker Path=s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/column_stats/col-stats-0001-0_0-0-0_00000000000000011.hfile.marker.APPEND 2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:202 - [direct] Created marker file s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/column_stats/col-stats-0001-0_0-0-0_00000000000000011.hfile.marker.APPEND in 12 ms 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:215 - Building HoodieLogFormat Writer 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:270 - HoodieLogFile on path s3a://warehouse/people/.hoodie/metadata/column_stats/.col-stats-0001-0_00000000000000011.log.1_0-0-0 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatWriter:122 - Append not supported.. Rolling over to HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/column_stats/.col-stats-0001-0_00000000000000011.log.2_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.io.HoodieAppendHandle:438 - AppendHandle for partitionPath column_stats filePath column_stats/.col-stats-0001-0_00000000000000011.log.2_0-0-0, took 43 ms. 2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:197 - Creating Marker Path=s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/column_stats/col-stats-0000-0_0-0-0_00000000000000011.hfile.marker.APPEND 2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:202 - [direct] Created marker file s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/column_stats/col-stats-0000-0_0-0-0_00000000000000011.hfile.marker.APPEND in 8 ms 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:215 - Building HoodieLogFormat Writer 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:270 - HoodieLogFile on path s3a://warehouse/people/.hoodie/metadata/column_stats/.col-stats-0000-0_00000000000000011.log.1_0-0-0 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatWriter:122 - Append not supported.. Rolling over to HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/column_stats/.col-stats-0000-0_00000000000000011.log.2_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.io.HoodieAppendHandle:438 - AppendHandle for partitionPath column_stats filePath column_stats/.col-stats-0000-0_00000000000000011.log.2_0-0-0, took 24 ms. 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.commit.BaseCommitActionExecutor:182 - Auto commit enabled: Committing 20240604232400230 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.CommitUtils:121 - Creating metadata for UPSERT_PREPPED numWriteStats:3 numReplaceFileIds:0 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor:205 - Committing 20240604232400230, action Type deltacommit 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor:211 - Committing 20240604232400230, action Type deltacommit 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:220 - Marking instant complete [==>20240604232400230__deltacommit__INFLIGHT] 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:615 - Checking for file exists ?s3a://warehouse/people/.hoodie/metadata/.hoodie/20240604232400230.deltacommit.inflight 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:623 - Create new file for toInstant ?s3a://warehouse/people/.hoodie/metadata/.hoodie/20240604232400230.deltacommit 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:224 - Completed [==>20240604232400230__deltacommit__INFLIGHT] 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor:219 - Committed 20240604232400230 2024-06-04 23:27:10 INFO org.apache.hudi.common.fs.FSUtils:716 - Removed directory at s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:220 - Marking instant complete [==>20240604232400230__replacecommit__INFLIGHT] 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:615 - Checking for file exists ?s3a://warehouse/people/.hoodie/20240604232400230.replacecommit.inflight 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:623 - Create new file for toInstant ?s3a://warehouse/people/.hoodie/20240604232400230.replacecommit 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:224 - Completed [==>20240604232400230__replacecommit__INFLIGHT] 2024-06-04 23:27:10 INFO org.apache.hudi.client.BaseHoodieWriteClient:238 - Committed 20240604232400230 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:165 - Creating InMemory based view for basePath s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 9 ms to read 1 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:109 - Load all partitions and files into file system view in advance. 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieTableMetadataUtil:927 - Loading latest merged file slices for metadata table partition files 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (files) 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:451 - Opened metadata base file from s3a://warehouse/people/.hoodie/metadata/files/files-0000-0_0-0-0_00000000000000010.hfile at instant 00000000000000010 in 3 ms 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:293 - Reading a delete block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatReader:115 - Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:286 - Reading a data block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0 at instant 20240604232400230 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:402 - Merging the final data blocks 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 2 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 1 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:206 - Number of log files scanned => 2 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:207 - MaxMemoryInBytes allowed for compaction => 1073741824 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:208 - Number of entries in MemoryBasedMap in ExternalSpillableMap => 6 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:209 - Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 5568 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:210 - Number of entries in DiskBasedMap in ExternalSpillableMap => 0 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:211 - Size of file spilled to disk => 0 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:501 - Opened 2 metadata log files (dataset instant=20240604232400230, metadata instant=20240604232400230) in 96 ms 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.BaseTableMetadata:334 - Listed partitions from metadata: #partitions=5 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:451 - Opened metadata base file from s3a://warehouse/people/.hoodie/metadata/files/files-0000-0_0-0-0_00000000000000010.hfile at instant 00000000000000010 in 0 ms 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:293 - Reading a delete block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatReader:115 - Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:286 - Reading a data block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0 at instant 20240604232400230 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:402 - Merging the final data blocks 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 2 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 1 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:206 - Number of log files scanned => 2 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:207 - MaxMemoryInBytes allowed for compaction => 1073741824 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:208 - Number of entries in MemoryBasedMap in ExternalSpillableMap => 6 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:209 - Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 5568 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:210 - Number of entries in DiskBasedMap in ExternalSpillableMap => 0 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:211 - Size of file spilled to disk => 0 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:501 - Opened 2 metadata log files (dataset instant=20240604232400230, metadata instant=20240604232400230) in 18 ms 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.BaseTableMetadata:397 - Listed files in 5 partitions from metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieTableMetadataUtil:927 - Loading latest merged file slices for metadata table partition files 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (files) 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:451 - Opened metadata base file from s3a://warehouse/people/.hoodie/metadata/files/files-0000-0_0-0-0_00000000000000010.hfile at instant 00000000000000010 in 0 ms 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:293 - Reading a delete block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatReader:115 - Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:286 - Reading a data block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0 at instant 20240604232400230 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:402 - Merging the final data blocks 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 2 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 1 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:206 - Number of log files scanned => 2 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:207 - MaxMemoryInBytes allowed for compaction => 1073741824 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:208 - Number of entries in MemoryBasedMap in ExternalSpillableMap => 6 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:209 - Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 5568 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:210 - Number of entries in DiskBasedMap in ExternalSpillableMap => 0 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:211 - Size of file spilled to disk => 0 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:501 - Opened 2 metadata log files (dataset instant=20240604232400230, metadata instant=20240604232400230) in 19 ms 2024-06-04 23:27:10 INFO org.apache.hudi.metadata.BaseTableMetadata:334 - Listed partitions from metadata: #partitions=5 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:317 - Cleaning city=DFW, retaining latest 0 commits. 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:501 - 0 patterns used to delete in partition path:city=DFW 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:317 - Cleaning city=NYC, retaining latest 0 commits. 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:501 - 0 patterns used to delete in partition path:city=NYC 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:317 - Cleaning city=ORD, retaining latest 0 commits. 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:501 - 0 patterns used to delete in partition path:city=ORD 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:317 - Cleaning city=SEA, retaining latest 0 commits. 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:501 - 0 patterns used to delete in partition path:city=SEA 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:317 - Cleaning city=SFO, retaining latest 0 commits. 2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:501 - 0 patterns used to delete in partition path:city=SFO 2024-06-04 23:27:10 WARN org.apache.hudi.client.HoodieTimelineArchiver:83 - The configured archival configs hoodie.keep.min.commits=0 is more aggressive than the cleaning configs as the earliest commit to retain is [20240604232400230__replacecommit__COMPLETED__20240604232710563]. Adjusted the archival configs to be hoodie.keep.min.commits=2 and hoodie.keep.max.commits=3 2024-06-04 23:27:10 WARN org.apache.hudi.client.HoodieTimelineArchiver:95 - Cleaning configs: hoodie.cleaner.policy=KEEP_LATEST_BY_HOURS hoodie.cleaner.hours.retained=168 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]} 2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups 2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations 2024-06-04 23:27:10 INFO org.apache.hudi.client.HoodieTimelineArchiver:514 - Not archiving as there is no compaction yet on the metadata table 2024-06-04 23:27:10 INFO org.apache.hudi.client.HoodieTimelineArchiver:177 - No Instants to archive 2024-06-04 23:27:10 INFO org.apache.iceberg.hadoop.HadoopTableOperations:164 - Committed a new metadata file s3a://warehouse/people/metadata/v1.metadata.json 2024-06-04 23:27:10 INFO org.apache.iceberg.SnapshotScan:120 - Scanning empty table s3a://warehouse/people 2024-06-04 23:27:11 INFO org.apache.iceberg.SnapshotProducer:414 - Committed snapshot 369315014296903406 (BaseOverwriteFiles) 2024-06-04 23:27:11 INFO org.apache.iceberg.metrics.LoggingMetricsReporter:38 - Received metrics report: CommitReport{tableName=s3a://warehouse/people, snapshotId=369315014296903406, sequenceNumber=1, operation=overwrite, commitMetrics=CommitMetricsResult{totalDuration=TimerResult{timeUnit=NANOSECONDS, totalDuration=PT0.085428125S, count=1}, attempts=CounterResult{unit=COUNT, value=1}, addedDataFiles=CounterResult{unit=COUNT, value=6}, removedDataFiles=null, totalDataFiles=CounterResult{unit=COUNT, value=6}, addedDeleteFiles=null, addedEqualityDeleteFiles=null, addedPositionalDeleteFiles=null, removedDeleteFiles=null, removedEqualityDeleteFiles=null, removedPositionalDeleteFiles=null, totalDeleteFiles=CounterResult{unit=COUNT, value=0}, addedRecords=CounterResult{unit=COUNT, value=6}, removedRecords=null, totalRecords=CounterResult{unit=COUNT, value=6}, addedFilesSizeInBytes=CounterResult{unit=BYTES, value=7621}, removedFilesSizeInBytes=null, totalFilesSizeInBytes=CounterResult{u nit=BYTES, value=7621}, addedPositionalDeletes=null, removedPositionalDeletes=null, totalPositionalDeletes=CounterResult{unit=COUNT, value=0}, addedEqualityDeletes=null, removedEqualityDeletes=null, totalEqualityDeletes=CounterResult{unit=COUNT, value=0}}, metadata={iceberg-version=Apache Iceberg 1.4.2 (commit f6bb9173b13424d77e7ad8439b5ef9627e530cb2)}} 2024-06-04 23:27:11 INFO org.apache.iceberg.RemoveSnapshots:124 - Expiring snapshots older than: 2024-05-28T23:27:11.069+00:00 (1716938831069) 2024-06-04 23:27:11 INFO org.apache.iceberg.RemoveSnapshots:309 - Committed snapshot changes 2024-06-04 23:27:11 INFO org.apache.iceberg.RemoveSnapshots:328 - Cleaning up expired files (local, incremental) 2024-06-04 23:27:11 INFO org.apache.iceberg.hadoop.HadoopTableOperations:164 - Committed a new metadata file s3a://warehouse/people/metadata/v2.metadata.json 2024-06-04 23:27:11 INFO org.apache.xtable.conversion.ConversionController:133 - Sync is successful for the following formats HUDI,ICEBERG ``` hudi-hive-sync/run_sync_tool.sh looks good ``` root@spark:/opt/hudi/hudi-sync/hudi-hive-sync# ./run_sync_tool.sh --metastore-uris 'thrift://hive-metastore:9083' --partitioned-by city --base-path 's3a://warehouse/people' --database hudi_db --table people --sync-mode hms ls: cannot access '/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-*.jar': No such file or directory setting hadoop conf dir Running Command : java -cp /hive/lib/hive-metastore-2.3.10.jar::/hive/lib/hive-service-2.3.10.jar::/hive/lib/hive-exec-2.3.10.jar::/hive/lib/hive-jdbc-2.3.10.jar:/hive/lib/hive-jdbc-handler-2.3.10.jar::/hive/lib/jackson-annotations-2.12.0.jar:/hive/lib/jackson-core-2.12.0.jar:/hive/lib/jackson-core-asl-1.9.13.jar:/hive/lib/jackson-databind-2.12.0.jar:/hive/lib/jackson-dataformat-smile-2.12.0.jar:/hive/lib/jackson-datatype-guava-2.12.0.jar:/hive/lib/jackson-datatype-joda-2.12.0.jar:/hive/lib/jackson-jaxrs-1.9.13.jar:/hive/lib/jackson-jaxrs-base-2.12.0.jar:/hive/lib/jackson-jaxrs-json-provider-2.12.0.jar:/hive/lib/jackson-jaxrs-smile-provider-2.12.0.jar:/hive/lib/jackson-mapper-asl-1.9.13.jar:/hive/lib/jackson-module-jaxb-annotations-2.12.0.jar:/hive/lib/jackson-module-scala_2.11-2.12.0.jar:/hive/lib/jackson-xc-1.9.13.jar::/hadoop/share/hadoop/common/*:/hadoop/share/hadoop/mapreduce/*:/hadoop/share/hadoop/hdfs/*:/hadoop/share/hadoop/common/lib/*:/hadoop/share/hadoop/hdfs/lib/*:/hado op/etc/hadoop: org.apache.hudi.hive.HiveSyncTool --metastore-uris thrift://hive-metastore:9083 --partitioned-by city --base-path s3a://warehouse/people --database hudi_db --table people --sync-mode hms SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/root/.ivy2/jars/org.slf4j_slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hadoop-2.10.2/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] 2024-06-04 23:30:49,554 INFO [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null 2024-06-04 23:30:49,765 INFO [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(133)) - Loading HoodieTableMetaClient from s3a://warehouse/people WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/root/.ivy2/jars/org.apache.hadoop_hadoop-auth-2.10.2.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2024-06-04 23:30:49,799 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-06-04 23:30:50,319 INFO [main] Configuration.deprecation (Configuration.java:logDeprecation(1294)) - fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key 2024-06-04 23:30:50,362 INFO [main] table.HoodieTableConfig (HoodieTableConfig.java:<init>(276)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties 2024-06-04 23:30:50,374 INFO [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(152)) - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people 2024-06-04 23:30:50,374 INFO [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(155)) - Loading Active commit timeline for s3a://warehouse/people 2024-06-04 23:30:50,392 INFO [main] timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(172)) - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]} 2024-06-04 23:30:50,427 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(408)) - Trying to connect to metastore with URI thrift://hive-metastore:9083 2024-06-04 23:30:50,435 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(493)) - Opened a connection to metastore, current connections: 1 2024-06-04 23:30:50,445 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(546)) - Connected to metastore. 2024-06-04 23:30:50,473 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(162)) - Syncing target hoodie table with hive table(hudi_db.people). Hive metastore URL from HiveConf:thrift://hive-metastore:9083). Hive metastore URL from HiveSyncConfig:thrift://hive-metastore:9083, basePath :s3a://warehouse/people 2024-06-04 23:30:50,473 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(224)) - Trying to sync hoodie table people with base path s3a://warehouse/people of type COPY_ON_WRITE 2024-06-04 23:30:50,659 INFO [main] table.TableSchemaResolver (TableSchemaResolver.java:readSchemaFromParquetBaseFile(329)) - Reading schema from s3a://warehouse/people/city=ORD/part-00003-6b980503-b6c2-42cb-aaa5-2d2feacda190.c000.snappy.parquet_20240604232400230_hudiext 2024-06-04 23:30:50,675 INFO [main] table.TableSchemaResolver (TableSchemaResolver.java:hasOperationField(470)) - Failed to read operation field from avro schema (Failed to read data schema) 2024-06-04 23:30:50,698 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncFirstTime(321)) - Sync table people for the first time. 2024-06-04 23:30:50,747 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(265)) - Last commit time synced was found to be null, last commit completion time is found to be null 2024-06-04 23:30:50,748 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(279)) - Sync all partitions given the last commit time synced is empty or before the start of the active timeline. Listing all partitions in s3a://warehouse/people, file system: S3AFileSystem{uri=s3a://warehouse, workingDir=s3a://warehouse/user/root, inputPolicy=normal, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, serverSideEncryptionAlgorithm='NONE', metastore=NullMetadataStore, authoritative=false, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=25, available=25, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@66813e6e[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], DirectoryMarkerRetention{policy='delete'}, statistics {31810 bytes read, 0 bytes written, 7 read ops, 0 large read ops, 0 wri te ops}, metrics {{Context=S3AFileSystem} {FileSystemId=3ce354be-808a-4240-813d-fc67413cd80a-warehouse} {fsURI=s3a://warehouse/people/.hoodie} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=6} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=1} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=4} {object_continue_list_requests=0} {object_metadata_requests=6} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {s3guard_metadatastore_put_pat h_request=5} {s3guard_metadatastore_initialization=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {S3guard_metadatastore_put_path_latencyNumOps=0} {S3guard_metadatastore_put_path_latency50thPercentileLatency=0} {S3guard_metadatastore_put_path_latency75thPercentileLatency=0} {S3guard_metadatastore_put_path_latency90thPercentileLatency=0} {S3guard_metadatastore_put_path_latency95thPercentileLatency=0} {S3guard_metadatastore_put_path_latency99thPercentileLatency=0} {stream_read_fully_operations=0} {stream_opened=3} {stream_bytes_skipped_on_seek=0} {stream_closed=3} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=31810} {stream_read_operations_incomplete=5} {stream_bytes_discarded_in_abort=0} {stream_close_operations=3} {stream_read_operations=33} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_se ek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }} 2024-06-04 23:30:50,780 INFO [main] table.HoodieTableConfig (HoodieTableConfig.java:<init>(276)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties 2024-06-04 23:30:50,906 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(298)) - Sync complete for people 2024-06-04 23:30:50,909 INFO [main] hive.metastore (HiveMetaStoreClient.java:close(576)) - Closed a connection to metastore, current connections: 0 ``` trino should show rows ``` trino@trino:/$ trino trino> select * from hudi.hudi_db.people; -> select * from delta.delta_db.people; -> select * from iceberg.iceberg_db.people; _hoodie_commit_time | _hoodie_commit_seqno | _hoodie_record_key | _hoodie_part> ---------------------+----------------------+--------------------+-------------> (0 rows) Query 20240604_233108_00000_ujp5k, FINISHED, 1 node Splits: 1 total, 1 done (100.00%) 1.36 [0 rows, 0B] [0 rows/s, 0B/s] id | name | age | city | create_ts ----+---------+-----+------+--------------------- 5 | Bob | 28 | SEA | 2023-09-23 00:00:00 6 | Charlie | 31 | DFW | 2023-08-29 00:00:00 4 | Andrew | 40 | NYC | 2023-10-28 00:00:00 1 | John | 25 | NYC | 2023-09-28 00:00:00 3 | Michael | 35 | ORD | 2023-09-28 00:00:00 2 | Emily | 30 | SFO | 2023-09-28 00:00:00 (6 rows) Query 20240604_233113_00001_ujp5k, FINISHED, 1 node Splits: 6 total, 6 done (100.00%) 0.83 [6 rows, 7.44KB] [7 rows/s, 8.96KB/s] id | name | age | city | create_ts ----+---------+-----+------+--------------------- 1 | John | 25 | NYC | 2023-09-28 00:00:00 4 | Andrew | 40 | NYC | 2023-10-28 00:00:00 2 | Emily | 30 | SFO | 2023-09-28 00:00:00 3 | Michael | 35 | ORD | 2023-09-28 00:00:00 6 | Charlie | 31 | DFW | 2023-08-29 00:00:00 5 | Bob | 28 | SEA | 2023-09-23 00:00:00 (6 rows) Query 20240604_233114_00002_ujp5k, FINISHED, 1 node Splits: 6 total, 6 done (100.00%) 0.95 [6 rows, 7.44KB] [6 rows/s, 7.8KB/s] ``` ### Are you willing to submit PR? - [X] I am willing to submit a PR! - [ ] I am willing to submit a PR but need help getting started! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
