This is an automated email from the ASF dual-hosted git repository.
codope pushed a change to branch release-1.0.0-beta2
in repository https://gitbox.apache.org/repos/asf/hudi.git
omit dda09b4e11a Ensure properties are copied when modifying schema (#11441)
omit ea2cb5c5d74 Create release branch for version 1.0.0-beta2
add 8eed6aefbc4 [MINOR] Update release guide (#11433)
add 3415a4a1a2c [HUDI-7819] Fix OptionsResolver#allowCommitOnEmptyBatch
default value (#11370)
add e38f78cedb9 [MINOR] Fix README for Slack link update and Linkedin
badge (#11442)
add 6f3539268f8 [HUDI-7854] Bump AWS SDK v2 version to 2.25.69 (#11426)
add eb20273cf66 Ensure properties are copied when modifying schema (#11441)
add 35d79270205 [MINOR] add `ad1happy2go` to github collaborators (#11447)
add 8f0467b7229 remove tableconfig from filegroup reader params (#11449)
add 35cdac0a645 [HUDI-7747] In MetaClient remove getBasePathV2() and
return StoragePath from getBasePath() (#11385)
add 6aea47a13f5 [HUDI-7671] Make Hudi timeline backward compatible (#11443)
add 84381afe7aa [HUDI-7872] Recreate and sync glue and hive table when
meta sync fails (#11451)
add 9b77eb1f5c6 [HUDI-7879] Optimize the redundant creation of HoodieTable
in DataSourceInternalWriterHelper and the unnecessary parameters in createTable
within BaseHoodieWriteClien (#11456)
add ec5244d5fe6 [HUDI-7880] Support extraMetadata in Spark SQL Insert Into
(#11458)
add b236396d877 [HUDI-7891] Fix
HoodieActiveTimeline#deleteCompletedRollback missing check for Action type
(#11462)
add ed0295f8fbd [HUDI-7847] Infer record merge mode during table upgrade
(#11439)
add 662f0822bd4 [HUDI-7838] Remove the option hoodie.schema.cache.enable
and always do the cache (#11444)
add bf1df335442 [HUDI-7876] use properties to store log spill map configs
for fg reader (#11455)
add 9f0130442a5 [HUDI-7874] Fix Hudi being able to read 2-level structure
(#11450)
add 1ce97bae116 [MINOR][DNM] Test disabling new HFile reader (#11488)
add 51c9c0e226a [HUDI-7906] Improve the parallelism deduce in rdd write
(#11470)
add 8a4bed03fa7 [HUDI-7849] Reduce time spent on running
testFiltersInFileFormat (#11423)
add 2cc45cc228a [HUDI-7881] Verify table base path as well for syncing
table in bigquery metastore (#11460)
add bb76de48e9f [HUDI-6508] Support compilation on Java 11 (#11479)
add b0580ef56ca [HUDI-5956] Fix spark DAG ui when write (#11376)
add 05a07cf76a0 [HUDI-7909] Add Comment to the FieldSchema returned by Aws
Glue Client (#11474)
add 7567eaef2c0 [HUDI-4123] Enchancing deltastreamer sql source tests
(#6781)
add dbc6ac50aec [HUDI-7395] Fix computation for metrics in
HoodieMetadataMetrics (#10641)
add 58b53f05980 [HUDI-4945] Add a test case for batch clean (#6845)
add 7178fa6d497 [MINOR] Adding tests for streaming read mor with
compaction (#6695)
add 6192cfb0e95 [MINOR] Reduce logging volume (#11505)
add c5ff6a2113f [MINOR] Removed useless checks from SqlBasedTransformers
(#11499)
add 4370178eb0b [HUDI-7927] Lazy init secondary view in FS view (#10652)
add 3152e47876f [MINOR] Bump JUnit version to 5.8.2 (#11511)
add 4b7e6e41573 [HUDI-7922] Add Hudi CLI bundle for Scala 2.13 (#11495)
add 1c731769d60 [HUDI-7882] Picking RFC-78 for bridge release (#11515)
add 7c2480f903b [HUDI-7403] Support Filter/Transformer to Hudi Exporter
Utility (non-hudi export) (#11509)
add ae1ee05ab8c [HUDI-7709] ClassCastException while reading the data
using `TimestampBasedKeyGenerator` (#11501)
add eb0725d1ef5 [HUDI-7877] Add record position to record index metadata
payload (#11467)
add c80b5596c1d [HUDI-7932] Fix Import ordering (#11524)
add ea6a6145223 [HUDI-7763] Fix that multiple jmx reporter can exist if
metadata enables (#11226)
add 8e36fe91715 [HUDI-7924] Capture Latency and Failure Metrics For Hive
Table recreation (#11498)
add 63b7b15fff2 [MINOR] doap: don't nest multiple versions in a release
(#11533)
add 4c0fb6d951e [HUDI-7914] Use internal schema without metadata fields
for delete-partitions OP (#11487)
add 107cffca4af [HUDI-7908] HoodieFileGroupReader fails if preCombine and
partition fields are the same (#11473)
add 78d7ee7cba5 Closing fsview in HoodieTableFileIndex (#11497)
add eeafa734a6c [HUDI-7903] Fix storage partition stats index to skip data
(#11472)
add 9f66ac0fb4b [HUDI-7942] Fix SmallFileAssignState#totalUnassigned
naming error (#11544)
add e942a27d446 [MINOR] Fix NPE after a clustering plan is finished
(#11550)
add 990c0c565b7 [HUDI-7933] Sync table in Glue/HMS if table base path is
updated (#11529)
add 9bc0164a387 [HUDI-7841] RLI and secondary index should consider only
pruned partitions for file skipping (#11434)
add a9a75f43a1c [HUDI-7941] add show_file_status procedure &
run_rollback_inflight_tableservice procedure (#11538)
add 29e2f6cd0f0 [HUDI-7926] Data skipping failure mode should be strict in
query test (#11502)
new bffdfa01172 Create release branch for version 1.0.0-beta2
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (dda09b4e11a)
\
N -- N -- N refs/heads/release-1.0.0-beta2 (bffdfa01172)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 2 +
.github/workflows/bot.yml | 169 +++++++++-
README.md | 5 +-
doap_HUDI.rdf | 42 +++
.../hudi/aws/cloudwatch/CloudWatchReporter.java | 12 +-
...dieConfigAWSAssumedRoleCredentialsProvider.java | 1 +
.../HoodieConfigAWSCredentialsProvider.java | 8 +-
.../hudi/aws/sync/AWSGlueCatalogSyncClient.java | 118 ++++++-
.../hudi/aws/sync/AwsGlueCatalogSyncTool.java | 6 +
.../hudi/config/GlueCatalogSyncClientConfig.java | 8 +
.../aws/cloudwatch/TestCloudWatchReporter.java | 10 +-
.../hudi/aws/sync/MockAwsGlueCatalogSyncTool.java | 17 +-
.../hudi/aws/sync/TestAWSGlueSyncClient.java | 257 ++++++++++++++
.../apache/hudi/aws/sync/TestAwsGlueSyncTool.java | 69 ++++
.../hudi/aws/sync/TestGluePartitionPushdown.java | 1 +
.../apache/hudi/aws/testutils/GlueTestUtil.java | 133 ++++++++
.../integ/ITTestDynamoDBBasedLockProvider.java | 17 +-
.../java/org/apache/hudi/cli/HoodiePrompt.java | 3 +-
.../hudi/cli/commands/ArchivedCommitsCommand.java | 10 +-
.../apache/hudi/cli/commands/BootstrapCommand.java | 4 +-
.../apache/hudi/cli/commands/CleansCommand.java | 6 +-
.../hudi/cli/commands/ClusteringCommand.java | 8 +-
.../hudi/cli/commands/CompactionCommand.java | 33 +-
.../apache/hudi/cli/commands/ExportCommand.java | 2 +-
.../hudi/cli/commands/HoodieLogFileCommand.java | 2 +-
.../cli/commands/HoodieSyncValidateCommand.java | 1 +
.../apache/hudi/cli/commands/MarkersCommand.java | 5 +-
.../apache/hudi/cli/commands/RepairsCommand.java | 13 +-
.../apache/hudi/cli/commands/RestoresCommand.java | 1 +
.../apache/hudi/cli/commands/RollbacksCommand.java | 3 +-
.../hudi/cli/commands/SavepointsCommand.java | 13 +-
.../apache/hudi/cli/commands/SparkEnvCommand.java | 1 +
.../org/apache/hudi/cli/commands/StatsCommand.java | 4 +-
.../org/apache/hudi/cli/commands/TableCommand.java | 3 +-
.../apache/hudi/cli/commands/TempViewCommand.java | 1 +
.../apache/hudi/cli/commands/TimelineCommand.java | 4 +-
.../cli/commands/UpgradeOrDowngradeCommand.java | 9 +-
.../org/apache/hudi/cli/commands/UtilsCommand.java | 1 +
.../java/org/apache/hudi/cli/TestSparkUtil.java | 4 +-
.../hudi/cli/commands/TestSparkEnvCommand.java | 2 +-
.../apache/hudi/cli/commands/TestTableCommand.java | 4 +-
.../testutils/HoodieCLIIntegrationTestBase.java | 1 +
.../testutils/HoodieCLIIntegrationTestHarness.java | 1 +
.../hudi/client/BaseHoodieTableServiceClient.java | 2 +-
.../apache/hudi/client/BaseHoodieWriteClient.java | 43 ++-
.../client/HoodieTableServiceManagerClient.java | 2 +-
.../client/heartbeat/HoodieHeartbeatClient.java | 2 +-
.../client/timeline/HoodieTimelineArchiver.java | 10 +-
.../client/transaction/ConcurrentOperation.java | 2 +-
.../hudi/client/transaction/lock/LockInfo.java | 3 +-
.../utils/LegacyArchivedMetaEntryReader.java | 2 +-
.../org/apache/hudi/config/HoodieIndexConfig.java | 2 +-
.../org/apache/hudi/config/HoodieWriteConfig.java | 25 +-
.../hudi/execution/CopyOnWriteInsertHandler.java | 1 +
.../hudi/execution/HoodieLazyInsertIterable.java | 3 +-
.../index/bucket/ConsistentBucketIndexUtils.java | 4 +-
.../hudi/index/bucket/HoodieSimpleBucketIndex.java | 11 +-
.../hudi/index/simple/HoodieGlobalSimpleIndex.java | 5 +-
.../hudi/index/simple/HoodieSimpleIndex.java | 10 +-
.../org/apache/hudi/io/HoodieAppendHandle.java | 2 +-
.../org/apache/hudi/io/HoodieBootstrapHandle.java | 5 +-
.../apache/hudi/io/HoodieMergeHandleFactory.java | 1 +
.../org/apache/hudi/io/HoodieMergedReadHandle.java | 2 +-
.../apache/hudi/io/HoodieSortedMergeHandle.java | 3 +-
.../io/HoodieSortedMergeHandleWithChangeLog.java | 2 +-
.../hudi/keygen/ComplexAvroKeyGenerator.java | 3 +-
.../hudi/keygen/GlobalAvroDeleteKeyGenerator.java | 3 +-
.../apache/hudi/keygen/SimpleAvroKeyGenerator.java | 3 +-
.../factory/HoodieAvroKeyGeneratorFactory.java | 2 +-
.../metadata/HoodieBackedTableMetadataWriter.java | 16 +-
.../hudi/metadata/HoodieMetadataWriteUtils.java | 7 +-
.../metadata/HoodieTableMetadataKeyGenerator.java | 3 +-
.../table/BucketIndexBulkInsertPartitioner.java | 1 +
.../java/org/apache/hudi/table/HoodieTable.java | 2 +-
.../hudi/table/action/clean/CleanPlanner.java | 2 +-
.../PartitionAwareClusteringPlanStrategy.java | 6 +-
.../util/ConsistentHashingUpdateStrategyUtils.java | 2 +-
.../table/action/commit/HoodieDeleteHelper.java | 2 +-
.../table/action/commit/HoodieWriteHelper.java | 2 +-
.../apache/hudi/table/action/commit/SmallFile.java | 3 +-
.../hudi/table/action/compact/CompactHelpers.java | 2 +-
.../hudi/table/action/compact/HoodieCompactor.java | 2 +-
.../BaseHoodieCompactionPlanGenerator.java | 10 +-
.../compact/strategy/CompactionStrategy.java | 2 +-
.../index/RecordBasedIndexingCatchupTask.java | 2 +-
.../table/action/index/RunIndexActionExecutor.java | 6 +-
.../table/action/rollback/BaseRollbackHelper.java | 2 +-
.../rollback/ListingBasedRollbackStrategy.java | 2 +-
.../rollback/MarkerBasedRollbackStrategy.java | 2 +-
.../hudi/table/marker/DirectWriteMarkers.java | 2 +-
.../marker/TimelineServerBasedWriteMarkers.java | 2 +-
.../hudi/table/marker/WriteMarkersFactory.java | 2 +-
.../table/upgrade/SevenToEightUpgradeHandler.java | 38 ++-
.../hudi/table/upgrade/UpgradeDowngrade.java | 2 +-
.../table/upgrade/ZeroToOneUpgradeHandler.java | 4 +-
.../java/org/apache/hudi/DummyActiveAction.java | 2 +-
.../heartbeat/TestHoodieHeartbeatClient.java | 14 +-
.../TestConflictResolutionStrategyUtil.java | 4 +-
.../table/timeline/TestHoodieArchivedTimeline.java | 2 +-
.../common/testutils/HoodieMetadataTestTable.java | 2 +-
...estCreateAvroKeyGeneratorByTypeWithFactory.java | 1 +
.../apache/hudi/metrics/TestHoodieJmxMetrics.java | 54 +++
.../datadog/TestHoodieMetricsDatadogConfig.java | 2 +-
.../apache/hudi/table/action/TestCleanPlanner.java | 1 -
.../table/action/commit/TestWriterHelperBase.java | 19 --
.../hudi/table/marker/TestWriteMarkersFactory.java | 3 +-
.../apache/hudi/table/repair/TestRepairUtils.java | 5 +-
.../hudi/utils/HoodieWriterClientTestHarness.java | 18 +-
.../hudi/utils/TestConcatenatingIterator.java | 1 +
.../apache/hudi/client/HoodieFlinkWriteClient.java | 5 +-
...inkSizeBasedClusteringPlanStrategyRecently.java | 2 +-
.../hudi/execution/FlinkLazyInsertIterable.java | 3 +-
.../commit/FlinkPartitionTTLActionExecutor.java | 1 +
.../hudi/table/action/commit/FlinkWriteHelper.java | 3 +-
.../apache/hudi/client/TestFlinkWriteClient.java | 2 +-
.../TestFlinkSizeBasedClusteringPlanStrategy.java | 1 +
.../testutils/HoodieFlinkWriteableTestTable.java | 2 +-
.../apache/hudi/client/HoodieJavaWriteClient.java | 5 +-
.../hudi/execution/JavaLazyInsertIterable.java | 3 +-
.../org/apache/hudi/table/HoodieJavaTable.java | 2 +-
.../JavaInsertPreppedCommitActionExecutor.java | 1 -
.../JavaUpsertPreppedCommitActionExecutor.java | 1 -
.../hudi/table/action/commit/JavaWriteHelper.java | 3 +-
.../client/TestHoodieJavaWriteClientInsert.java | 2 +-
.../apache/hudi/client/SparkRDDWriteClient.java | 5 +-
.../run/strategy/ExecutionStrategyUtil.java | 9 +-
.../hudi/client/model/HoodieInternalRow.java | 1 +
.../org/apache/hudi/data/HoodieJavaPairRDD.java | 23 ++
.../java/org/apache/hudi/data/HoodieJavaRDD.java | 23 ++
.../hudi/execution/SparkLazyInsertIterable.java | 2 +-
.../bulkinsert/RDDBucketIndexPartitioner.java | 7 +-
.../RDDSimpleBucketBulkInsertPartitioner.java | 1 +
.../bulkinsert/RDDSpatialCurveSortPartitioner.java | 7 +-
.../bulkinsert/RowSpatialCurveSortPartitioner.java | 1 +
.../index/bloom/SparkHoodieBloomIndexHelper.java | 4 +-
.../hudi/io/storage/HoodieSparkFileWriter.java | 3 +-
.../io/storage/HoodieSparkParquetStreamWriter.java | 10 +-
.../storage/row/HoodieRowParquetWriteSupport.java | 2 +-
.../apache/hudi/keygen/ComplexKeyGenerator.java | 3 +-
.../org/apache/hudi/keygen/SimpleKeyGenerator.java | 3 +-
.../hudi/keygen/SparkKeyGeneratorInterface.java | 1 +
.../hudi/keygen/TimestampBasedKeyGenerator.java | 3 +-
.../keygen/UTF8StringPartitionPathFormatter.java | 1 +
.../SparkHoodieBackedTableMetadataWriter.java | 2 +-
.../SparkHoodieMetadataBulkInsertPartitioner.java | 12 +-
.../apache/hudi/metrics/DistributedRegistry.java | 9 +-
.../org/apache/hudi/table/HoodieSparkTable.java | 2 +-
.../BucketBulkInsertDataInternalWriterHelper.java | 1 +
.../action/commit/SparkBucketIndexPartitioner.java | 26 +-
.../action/commit/SparkHoodiePartitioner.java | 1 +
.../scala/org/apache/hudi/HoodieSparkUtils.scala | 4 +-
.../scala/org/apache/hudi/HoodieUnsafeRDD.scala | 1 -
.../org/apache/hudi/SparkConversionUtils.scala | 1 +
.../SparkFileFormatInternalRowReaderContext.scala | 7 +-
.../hudi/util/CloseableInternalRowIterator.scala | 1 +
.../org/apache/hudi/util/JavaScalaConverters.scala | 8 +
.../apache/spark/sql/BucketPartitionUtils.scala | 1 +
.../scala/org/apache/spark/sql/DataFrameUtil.scala | 1 +
.../spark/sql/HoodieCatalystExpressionUtils.scala | 1 +
.../org/apache/spark/sql/HoodieUnsafeUtils.scala | 1 +
.../spark/sql/execution/SQLConfInjectingRDD.scala | 2 +-
.../datasources/SparkParsePartitionUtil.scala | 5 +-
.../datasources/parquet/SparkParquetReader.scala | 3 +-
.../org/apache/spark/sql/hudi/SparkAdapter.scala | 9 +
.../spark/sql/hudi/execution/RangeSample.scala | 1 +
.../org/apache/hudi/client/TestClientRollback.java | 8 +-
.../hudi/client/TestPartitionTTLManagement.java | 6 +-
...stSparkHoodieMetadataBulkInsertPartitioner.java | 11 +-
.../hudi/client/TestSparkRDDWriteClient.java | 4 +-
.../TestUniformBootstrapModeSelector.java | 1 +
.../TestSparkSizeBasedClusteringPlanStrategy.java | 1 -
.../functional/TestExternalPathHandling.java | 4 +-
.../TestHoodieClientOnCopyOnWriteStorage.java | 12 +-
.../TestRemoteFileSystemViewWithMetadataTable.java | 2 +-
.../org/apache/hudi/data/TestHoodieJavaRDD.java | 28 ++
.../TestBoundedInMemoryExecutorInSpark.java | 2 +-
.../hudi/execution/TestBoundedInMemoryQueue.java | 2 +-
.../execution/TestDisruptorExecutionInSpark.java | 1 +
.../hudi/execution/TestDisruptorMessageQueue.java | 8 +-
.../hudi/execution/TestSimpleExecutionInSpark.java | 2 +-
.../index/hbase/TestHBaseQPSResourceAllocator.java | 2 +-
.../java/org/apache/hudi/table/TestCleaner.java | 2 +-
.../apache/hudi/table/TestConsistencyGuard.java | 2 +-
.../hudi/table/TestHoodieMergeOnReadTable.java | 8 +-
.../table/action/cluster/ClusteringTestUtils.java | 2 +-
.../commit/TestCopyOnWriteActionExecutor.java | 15 +-
.../table/action/commit/TestSparkWriteHelper.java | 23 ++
.../table/action/rollback/TestRollbackUtils.java | 2 +-
...dieSparkMergeOnReadTableInsertUpdateDelete.java | 3 +-
.../hudi/table/marker/TestDirectWriteMarkers.java | 2 +-
.../TestTimelineServerBasedWriteMarkers.java | 2 +-
.../hudi/table/upgrade/TestUpgradeDowngrade.java | 12 +-
.../testutils/HoodieSparkClientTestHarness.java | 6 +
.../testutils/HoodieSparkWriteableTestTable.java | 6 +-
hudi-common/src/main/avro/HoodieMetadata.avsc | 9 +
.../org/apache/hudi/BaseHoodieTableFileIndex.java | 75 +++--
.../hudi/common/config/HoodieReaderConfig.java | 2 +-
.../DirectMarkerBasedDetectionStrategy.java | 4 +-
.../org/apache/hudi/common/data/HoodieData.java | 5 +
.../apache/hudi/common/data/HoodieListData.java | 5 +
.../hudi/common/data/HoodieListPairData.java | 5 +
.../apache/hudi/common/data/HoodiePairData.java | 5 +
.../java/org/apache/hudi/common/fs/FSUtils.java | 14 +
.../hudi/common/fs/OptimisticConsistencyGuard.java | 2 +-
.../hudi/common/model/BootstrapIndexType.java | 2 +-
.../hudi/common/model/FirstValueAvroPayload.java | 5 +-
.../hudi/common/model/HoodieAvroRecordMerger.java | 6 +-
.../apache/hudi/common/model/HoodieLogFile.java | 4 +-
.../hudi/common/model/HoodieRecordPayload.java | 5 +-
.../common/model/PartialUpdateAvroPayload.java | 10 +-
.../hudi/common/table/HoodieTableConfig.java | 6 +-
.../hudi/common/table/HoodieTableMetaClient.java | 44 +--
.../hudi/common/table/TableSchemaResolver.java | 10 +-
.../hudi/common/table/cdc/HoodieCDCExtractor.java | 4 +-
.../table/log/AbstractHoodieLogRecordReader.java | 12 +-
.../log/BaseHoodieMergedLogRecordScanner.java | 9 +-
.../table/log/HoodieMergedLogRecordScanner.java | 6 +
.../log/HoodieMetadataMergedLogRecordScanner.java | 6 +
.../table/log/HoodieUnMergedLogRecordScanner.java | 7 +
.../read/HoodieBaseFileGroupRecordBuffer.java | 49 +--
.../common/table/read/HoodieFileGroupReader.java | 15 +-
.../read/HoodieKeyBasedFileGroupRecordBuffer.java | 10 +-
.../HoodiePositionBasedFileGroupRecordBuffer.java | 10 +-
.../read/HoodieUnmergedFileGroupRecordBuffer.java | 10 +-
.../table/timeline/HoodieActiveTimeline.java | 1 +
.../hudi/common/table/timeline/HoodieInstant.java | 5 +-
.../table/timeline/HoodieInstantTimeGenerator.java | 6 +
.../hudi/common/table/timeline/TimelineUtils.java | 2 +-
.../clean/CleanMetadataV1MigrationHandler.java | 2 +-
.../clean/CleanPlanV2MigrationHandler.java | 2 +-
.../compaction/CompactionV1MigrationHandler.java | 2 +-
.../table/view/AbstractTableFileSystemView.java | 8 +-
.../common/table/view/FileSystemViewManager.java | 72 ++--
.../view/HoodieTablePreCommitFileSystemView.java | 2 +-
.../IncrementalTimelineSyncFileSystemView.java | 6 +-
.../table/view/PriorityBasedFileSystemView.java | 90 ++---
.../view/RemoteHoodieTableFileSystemView.java | 2 +-
.../table/view/RocksDbBasedFileSystemView.java | 4 +-
.../common/table/view/SyncableFileSystemView.java | 2 +-
.../common/util/HoodieCommonKryoRegistrar.java | 5 +-
.../hudi/common/util/InternalSchemaCache.java | 14 +-
.../hudi/common/util/RocksDBSchemaHelper.java | 15 +-
.../hudi/common/util/SerializationUtils.java | 3 +-
.../hudi/common/util/collection/FlatLists.java | 10 +-
.../common/util/queue/DisruptorMessageQueue.java | 2 +-
.../common/util/queue/WaitStrategyFactory.java | 3 +-
.../java/org/apache/hudi/expression/Literal.java | 1 +
.../apache/hudi/io/storage/HoodieFileWriter.java | 4 +-
.../hudi/io/storage/HoodieSeekingFileReader.java | 3 +-
.../apache/hudi/metadata/BaseTableMetadata.java | 9 +-
.../metadata/HoodieMetadataFileSystemView.java | 2 +-
.../metadata/HoodieMetadataLogRecordReader.java | 6 +
.../hudi/metadata/HoodieMetadataMetrics.java | 9 +-
.../hudi/metadata/HoodieMetadataPayload.java | 16 +-
.../apache/hudi/metadata/HoodieTableMetadata.java | 4 +
.../hudi/metadata/HoodieTableMetadataUtil.java | 57 ++--
.../apache/hudi/metrics/JmxMetricsReporter.java | 36 +-
.../java/org/apache/hudi/metrics/MetricUtils.java | 1 +
.../apache/hudi/metrics/m3/M3MetricsReporter.java | 14 +-
.../hudi/metrics/m3/M3ScopeReporterAdaptor.java | 4 +-
.../main/java/org/apache/hudi/util/Transient.java | 1 +
.../common/model/TestFirstValueAvroPayload.java | 3 +-
.../model/TestHoodieLSMTimelineManifest.java | 2 +-
.../table/read/TestHoodieFileGroupReaderBase.java | 12 +-
.../common/table/timeline/TestLSMTimeline.java | 2 +-
.../view/TestPriorityBasedFileSystemView.java | 83 ++++-
.../hudi/common/testutils/CompactionTestUtils.java | 2 +-
.../hudi/common/testutils/HoodieTestUtils.java | 6 +
.../hudi/common/testutils/SchemaTestUtil.java | 35 ++
.../reader/HoodieFileGroupReaderTestUtils.java | 24 +-
.../hudi/common/util/TestHoodieRecordUtils.java | 3 +-
.../apache/hudi/common/util/TestRetryHelper.java | 2 +-
.../hudi/expression/TestPartialBindVisitor.java | 1 +
...compatible.avsc => complex-schema-evolved.avsc} | 7 +-
.../src/test/resources/complex-schema-evolved.data | 41 +++
hudi-examples/hudi-examples-common/pom.xml | 14 -
.../examples/quickstart/HoodieFlinkQuickstart.java | 25 +-
.../factory/ContinuousFileSourceFactory.java | 5 +-
.../quickstart/utils/QuickstartConfigurations.java | 22 +-
.../quickstart/TestHoodieFlinkQuickstart.java | 3 +-
hudi-examples/hudi-examples-java/pom.xml | 14 -
.../examples/common/ExampleDataSchemaProvider.java | 3 +-
.../hudi/examples/common/IdentityTransformer.java | 1 +
.../examples/spark/HoodieDataSourceExample.scala | 1 +
.../examples/spark/HoodieMorCompactionJob.scala | 1 +
.../apache/hudi/configuration/OptionsResolver.java | 2 +-
.../org/apache/hudi/sink/StreamWriteFunction.java | 2 +-
.../hudi/sink/bootstrap/BootstrapOperator.java | 3 +-
.../sink/bucket/BucketStreamWriteFunction.java | 6 +-
.../bucket/ConsistentBucketAssignFunction.java | 3 +-
.../apache/hudi/sink/bulk/sort/SortOperator.java | 3 +-
.../hudi/sink/partitioner/BucketAssigner.java | 6 +-
.../sink/partitioner/BucketIndexPartitioner.java | 2 +-
.../apache/hudi/source/ExpressionPredicates.java | 3 +-
.../apache/hudi/source/IncrementalInputSplits.java | 4 +-
.../org/apache/hudi/table/HoodieTableSource.java | 8 +-
.../hudi/table/catalog/HoodieHiveCatalog.java | 8 +-
.../hudi/table/format/InternalSchemaManager.java | 2 +-
.../apache/hudi/table/format/RecordIterators.java | 2 +-
.../table/format/mor/MergeOnReadInputFormat.java | 4 +-
.../apache/hudi/util/RowDataCastProjection.java | 1 +
.../org/apache/hudi/util/RowDataProjection.java | 1 +
.../java/org/apache/hudi/util/StreamerUtil.java | 8 +-
.../sink/TestStreamWriteOperatorCoordinator.java | 16 +-
.../org/apache/hudi/sink/TestWriteCopyOnWrite.java | 10 +-
.../hudi/sink/TestWriterWithPartitionTTl.java | 3 +-
.../ITTestFlinkConsistentHashingClustering.java | 2 +-
.../sink/cluster/ITTestHoodieFlinkClustering.java | 2 +-
.../sink/compact/ITTestHoodieFlinkCompactor.java | 2 +-
.../apache/hudi/sink/compact/TestCustomSerDe.java | 17 +-
.../org/apache/hudi/sink/utils/TestWriteBase.java | 14 +-
.../hudi/source/TestExpressionEvaluators.java | 3 +-
.../hudi/source/TestExpressionPredicates.java | 4 +-
.../apache/hudi/table/ITTestHoodieDataSource.java | 30 +-
.../apache/hudi/table/format/TestInputFormat.java | 14 +-
.../table/format/cow/TestBlockLocationSort.java | 9 +-
.../org/apache/hudi/utils/FlinkMiniCluster.java | 2 -
.../org/apache/hudi/utils/TestClusteringUtil.java | 3 +-
.../hudi/utils/TestRowDataToAvroConverters.java | 8 +-
.../test/java/org/apache/hudi/utils/TestUtils.java | 2 +-
.../gcp/bigquery/HoodieBigQuerySyncClient.java | 15 +-
.../hudi/gcp/bigquery/TestBigQuerySyncTool.java | 5 +
.../gcp/bigquery/TestBigQuerySyncToolArgs.java | 2 +-
.../gcp/bigquery/TestHoodieBigQuerySyncClient.java | 40 ++-
.../org/apache/hudi/common/util/ParquetUtils.java | 21 +-
.../hadoop/fs/HoodieSerializableFileStatus.java | 2 +-
.../hudi/io/hadoop/HoodieAvroParquetReader.java | 2 +-
.../apache/parquet/avro/HoodieAvroReadSupport.java | 23 +-
.../common/table/TestHoodieTableMetaClient.java | 2 +-
.../table/timeline/TestHoodieActiveTimeline.java | 28 ++
.../common/table/timeline/TestHoodieInstant.java | 1 +
.../view/TestHoodieTableFSViewWithClustering.java | 2 +-
.../table/view/TestHoodieTableFileSystemView.java | 2 +-
.../table/view/TestIncrementalFSViewSync.java | 12 +-
.../common/testutils/HoodieCommonTestHarness.java | 4 +-
.../hudi/common/testutils/HoodieTestTable.java | 4 +-
.../reader/HoodieFileGroupReaderTestHarness.java | 16 +-
.../common/util/collection/TestBitCaskDiskMap.java | 2 +-
.../util/collection/TestExternalSpillableMap.java | 2 +-
.../common/util/collection/TestRocksDbDiskMap.java | 2 +-
.../TestFileSystemBackedTableMetadata.java | 2 +-
.../hudi/metadata/TestHoodieTableMetadataUtil.java | 2 +-
.../hudi/hadoop/HiveHoodieTableFileIndex.java | 2 +-
.../hadoop/HoodieCopyOnWriteTableInputFormat.java | 3 +-
.../HoodieFileGroupReaderBasedRecordReader.java | 24 +-
.../apache/hudi/hadoop/HoodieHFileInputFormat.java | 5 +-
.../hudi/hadoop/HoodieParquetInputFormatBase.java | 3 +-
.../hudi/hadoop/HoodieROTablePathFilter.java | 2 +-
.../org/apache/hudi/hadoop/InputPathHandler.java | 2 +-
.../org/apache/hudi/hadoop/RealtimeFileStatus.java | 5 +-
.../apache/hudi/hadoop/SchemaEvolutionContext.java | 20 +-
.../hive/HoodieCombineRealtimeHiveSplit.java | 8 +-
.../HoodieMergeOnReadTableInputFormat.java | 4 +-
.../HoodieRealtimeBootstrapBaseFileSplit.java | 3 +-
.../hudi/hadoop/realtime/HoodieRealtimePath.java | 3 +-
.../hudi/hadoop/utils/HiveAvroSerializer.java | 1 -
.../apache/hudi/hadoop/utils/HoodieHiveUtils.java | 1 -
.../hudi/hadoop/utils/HoodieInputFormatUtils.java | 8 +-
.../org/apache/hudi/hadoop/TestAnnotation.java | 4 +-
.../hudi/hadoop/utils/TestHiveAvroSerializer.java | 2 +-
.../org/apache/hudi/common/util/StringUtils.java | 25 ++
.../apache/hudi/common/util/TestStringUtils.java | 10 +
.../connect/TestConnectTransactionParticipant.java | 2 +-
.../apache/hudi/connect/TestHdfsConfiguration.java | 7 +-
.../main/java/org/apache/hudi/DataSourceUtils.java | 2 +-
.../hudi/HoodieSparkFunctionalIndexClient.java | 4 +-
.../BaseDatasetBulkInsertCommitActionExecutor.java | 1 +
.../DatasetBulkInsertCommitActionExecutor.java | 1 +
...setBulkInsertOverwriteCommitActionExecutor.java | 1 +
...lkInsertOverwriteTableCommitActionExecutor.java | 2 +-
.../internal/DataSourceInternalWriterHelper.java | 4 +-
.../scala/org/apache/hudi/AvroProjection.scala | 3 +-
.../org/apache/hudi/BaseFileOnlyRelation.scala | 2 +-
.../scala/org/apache/hudi/BucketIndexSupport.scala | 9 +-
.../org/apache/hudi/ColumnStatsIndexSupport.scala | 2 +-
.../main/scala/org/apache/hudi/DefaultSource.scala | 3 -
.../scala/org/apache/hudi/FileSystemRelation.scala | 2 +-
.../scala/org/apache/hudi/HoodieBaseRelation.scala | 9 +-
.../org/apache/hudi/HoodieBootstrapMORRDD.scala | 4 +-
.../apache/hudi/HoodieBootstrapMORRelation.scala | 1 -
.../scala/org/apache/hudi/HoodieBootstrapRDD.scala | 1 +
.../org/apache/hudi/HoodieBootstrapRelation.scala | 3 +-
.../scala/org/apache/hudi/HoodieCLIUtils.scala | 4 +-
.../org/apache/hudi/HoodieCatalystUtils.scala | 1 +
.../org/apache/hudi/HoodieCreateRecordUtils.scala | 3 +-
.../scala/org/apache/hudi/HoodieFileIndex.scala | 19 +-
.../hudi/HoodieHadoopFsRelationFactory.scala | 13 +-
.../apache/hudi/HoodieIncrementalFileIndex.scala | 4 +-
.../org/apache/hudi/HoodieMergeOnReadRDD.scala | 5 +-
.../hudi/HoodiePartitionCDCFileGroupMapping.scala | 1 -
.../hudi/HoodiePartitionFileSliceMapping.scala | 3 -
.../scala/org/apache/hudi/HoodieSchemaUtils.scala | 1 +
.../org/apache/hudi/HoodieSparkConfUtils.scala | 1 +
.../org/apache/hudi/HoodieSparkSqlWriter.scala | 64 ++--
.../org/apache/hudi/IncrementalRelation.scala | 6 +-
.../hudi/MergeOnReadIncrementalRelation.scala | 9 +-
.../apache/hudi/MergeOnReadSnapshotRelation.scala | 1 -
.../apache/hudi/PartitionStatsIndexSupport.scala | 4 +-
.../org/apache/hudi/RecordLevelIndexSupport.scala | 46 ++-
.../org/apache/hudi/SecondaryIndexSupport.scala | 7 +-
.../org/apache/hudi/SparkBaseIndexSupport.scala | 22 +-
.../apache/hudi/SparkHoodieTableFileIndex.scala | 10 +-
.../org/apache/hudi/cdc/CDCFileGroupIterator.scala | 2 +-
.../scala/org/apache/hudi/cdc/CDCRelation.scala | 2 +-
.../scala/org/apache/hudi/cdc/HoodieCDCRDD.scala | 4 +-
.../cdc/InternalRowToJsonStringConverter.scala | 3 +-
.../org/apache/spark/HoodieHadoopFSUtils.scala | 3 +-
.../datasources/HoodieInMemoryFileIndex.scala | 5 +-
.../sql/FileFormatUtilsForFileGroupReader.scala | 6 +-
.../apache/spark/sql/avro/SchemaConverters.scala | 4 +-
.../sql/catalyst/catalog/HoodieCatalogTable.scala | 8 +-
.../datasources/HoodieMultipleBaseFileFormat.scala | 9 +-
...odieFileGroupReaderBasedParquetFileFormat.scala | 28 +-
.../parquet/LegacyHoodieParquetFileFormat.scala | 4 +-
.../apache/spark/sql/hudi/DataSkippingUtils.scala | 4 +-
.../spark/sql/hudi/HoodieSqlCommonUtils.scala | 23 +-
.../spark/sql/hudi/ProvidesHoodieConfig.scala | 14 +-
.../AlterHoodieTableAddColumnsCommand.scala | 6 +-
.../AlterHoodieTableAddPartitionCommand.scala | 1 -
.../AlterHoodieTableDropPartitionCommand.scala | 2 +-
.../command/AlterHoodieTableRenameCommand.scala | 3 +-
.../hudi/command/CreateHoodieTableCommand.scala | 15 +-
.../sql/hudi/command/DropHoodieTableCommand.scala | 3 +-
...odieSparkValidateDuplicateKeyRecordMerger.scala | 2 +-
.../hudi/command/RepairHoodieTableCommand.scala | 7 +-
.../command/ShowHoodieTablePartitionsCommand.scala | 1 -
.../hudi/command/TruncateHoodieTableCommand.scala | 5 +-
.../hudi/command/ValidateDuplicateKeyPayload.scala | 1 -
.../sql/hudi/streaming/HoodieMetadataLog.scala | 6 +-
.../sql/hudi/streaming/HoodieSourceOffset.scala | 1 -
.../sql/hudi/streaming/HoodieStreamSource.scala | 11 +-
.../datasources/TestHoodieInMemoryFileIndex.scala | 3 +-
.../spark/sql/hive/TestHiveClientUtils.scala | 2 +-
.../spark/sql/hudi/TestProvidesHoodieConfig.scala | 1 -
.../main/java/org/apache/hudi/QuickstartUtils.java | 7 +-
.../org/apache/hudi/payload/AWSDmsAvroPayload.java | 3 +-
.../org/apache/spark/sql/hudi/DedupeSparkJob.scala | 2 +-
.../sql/hudi/HoodieSparkSessionExtension.scala | 3 +-
.../spark/sql/hudi/analysis/HoodieAnalysis.scala | 2 +
.../analysis/HoodiePruneFileSourcePartitions.scala | 1 +
.../hudi/command/CallProcedureHoodieCommand.scala | 1 +
.../command/CompactionHoodieTableCommand.scala | 1 +
.../command/CompactionShowHoodieTableCommand.scala | 1 +
.../command/CreateHoodieTableAsSelectCommand.scala | 5 +-
.../hudi/command/DeleteHoodieTableCommand.scala | 3 +-
.../spark/sql/hudi/command/IndexCommands.scala | 1 +
.../command/InsertIntoHoodieTableCommand.scala | 1 +
.../hudi/command/MergeIntoHoodieTableCommand.scala | 4 +-
.../hudi/command/UpdateHoodieTableCommand.scala | 3 +-
.../spark/sql/hudi/command/UuidKeyGenerator.scala | 6 +-
.../hudi/command/payload/ExpressionPayload.scala | 8 +-
.../procedures/ArchiveCommitsProcedure.scala | 1 +
.../procedures/CommitsCompareProcedure.scala | 1 +
.../command/procedures/CopyToTableProcedure.scala | 1 +
.../procedures/CopyToTempViewProcedure.scala | 1 +
.../command/procedures/DeleteMarkerProcedure.scala | 2 +
.../procedures/ExportInstantsProcedure.scala | 3 +-
.../procedures/HdfsParquetImportProcedure.scala | 2 +
.../hudi/command/procedures/HelpProcedure.scala | 1 +
.../command/procedures/HiveSyncProcedure.scala | 3 +-
.../command/procedures/HoodieProcedureUtils.scala | 1 +
.../hudi/command/procedures/HoodieProcedures.scala | 2 +
.../sql/hudi/command/procedures/Procedure.scala | 4 -
.../RepairAddpartitionmetaProcedure.scala | 1 +
.../RepairCorruptedCleanFilesProcedure.scala | 1 +
.../procedures/RepairDeduplicateProcedure.scala | 1 +
.../RollbackToInstantTimeProcedure.scala | 1 -
.../command/procedures/RunBootstrapProcedure.scala | 2 +-
.../command/procedures/RunCleanProcedure.scala | 2 +-
.../procedures/RunClusteringProcedure.scala | 3 +-
.../procedures/RunCompactionProcedure.scala | 1 +
.../RunRollbackInflightTableServiceProcedure.scala | 143 ++++++++
.../hudi/command/procedures/RunTTLProcedure.scala | 2 +
.../procedures/ShowArchivedCommitsProcedure.scala | 1 +
.../procedures/ShowBootstrapMappingProcedure.scala | 1 +
.../procedures/ShowClusteringProcedure.scala | 1 +
.../procedures/ShowCommitFilesProcedure.scala | 1 +
.../procedures/ShowCommitPartitionsProcedure.scala | 1 +
.../procedures/ShowCommitWriteStatsProcedure.scala | 1 +
.../command/procedures/ShowCommitsProcedure.scala | 1 +
.../procedures/ShowCompactionProcedure.scala | 1 +
.../procedures/ShowFileStatusProcedure.scala | 244 ++++++++++++++
.../procedures/ShowFileSystemViewProcedure.scala | 2 +-
.../procedures/ShowFsPathDetailProcedure.scala | 1 -
.../ShowHoodieLogFileMetadataProcedure.scala | 3 +-
.../ShowHoodieLogFileRecordsProcedure.scala | 4 +-
.../ShowMetadataTablePartitionsProcedure.scala | 1 -
.../procedures/ShowTablePropertiesProcedure.scala | 1 +
.../procedures/StatsFileSizeProcedure.scala | 1 +
.../StatsWriteAmplificationProcedure.scala | 1 +
.../procedures/UpgradeOrDowngradeProcedure.scala | 5 +-
.../procedures/ValidateHoodieSyncProcedure.scala | 1 +
.../spark/sql/parser/HoodieCommonSqlParser.scala | 11 +-
.../sql/parser/HoodieSqlCommonAstBuilder.scala | 6 +-
...stHoodiePositionBasedFileGroupRecordBuffer.java | 13 +-
.../java/org/apache/hudi/TestQuickstartUtils.java | 1 +
.../TestDataSkippingWithMORColstats.java | 2 +-
.../functional/TestFiltersInFileGroupReader.java | 4 +-
.../TestGlobalIndexEnableUpdatePartitions.java | 2 +-
.../apache/hudi/functional/TestOrcBootstrap.java | 4 +-
.../hudi/keygen/TestComplexKeyGenerator.java | 3 +-
.../keygen/TestGlobalDeleteRecordGenerator.java | 3 +-
.../keygen/TestNonpartitionedKeyGenerator.java | 3 +-
.../apache/hudi/keygen/TestSimpleKeyGenerator.java | 3 +-
.../TestCreateKeyGeneratorByTypeWithFactory.java | 1 +
.../hudi/testutils/HoodieSparkClientTestBase.java | 1 +
.../apache/hudi/HoodieSparkWriterTestBase.scala | 3 +-
.../scala/org/apache/hudi/SparkDatasetMixin.scala | 3 +-
.../org/apache/hudi/TestAvroConversionUtils.scala | 5 +-
.../hudi/TestAvroSchemaResolutionSupport.scala | 1 +
.../org/apache/hudi/TestDataSkippingUtils.scala | 4 +-
.../org/apache/hudi/TestDataSourceDefaults.scala | 7 +-
.../org/apache/hudi/TestDataSourceOptions.scala | 1 +
.../hudi/TestGenericRecordAndRowConsistency.scala | 3 +-
.../org/apache/hudi/TestHoodieFileIndex.scala | 4 +-
.../org/apache/hudi/TestHoodieParquetBloom.scala | 12 +-
.../org/apache/hudi/TestHoodieRelations.scala | 3 +-
.../org/apache/hudi/TestHoodieSparkSqlWriter.scala | 2 +-
.../TestHoodieSparkSqlWriterPartitionTTL.scala | 5 +-
.../apache/hudi/TestHoodieSparkSqlWriterUtc.scala | 3 +-
.../org/apache/hudi/TestHoodieSparkUtils.scala | 3 +-
.../hudi/TestParquetReaderCompatibility.scala | 327 ++++++++++++++++++
.../model/TestHoodieRecordSerialization.scala | 3 +-
.../read/TestHoodieFileGroupReaderOnSpark.scala | 8 +-
.../TestSpark35RecordPositionMetadataColumn.scala | 3 +-
.../hudi/functional/ColumnStatIndexTestBase.scala | 2 +
.../apache/hudi/functional/CommonOptionUtils.scala | 2 +-
.../functional/PartitionStatsIndexTestBase.scala | 1 +
.../hudi/functional/RecordLevelIndexTestBase.scala | 35 +-
.../hudi/functional/SecondaryIndexTestBase.scala | 2 +
.../functional/TestBloomFiltersIndexSupport.scala | 3 +
.../hudi/functional/TestBucketIndexSupport.scala | 9 +-
.../apache/hudi/functional/TestCOWDataSource.scala | 2 +-
.../functional/TestColumnStatsIndexWithSQL.scala | 2 +-
.../functional/TestDataSourceForBootstrap.scala | 5 +-
.../apache/hudi/functional/TestEmptyCommit.scala | 4 +-
.../hudi/functional/TestHoodieActiveTimeline.scala | 1 -
.../hudi/functional/TestLayoutOptimization.scala | 1 -
.../hudi/functional/TestMORDataSourceStorage.scala | 1 +
.../hudi/functional/TestMetadataRecordIndex.scala | 4 +-
.../hudi/functional/TestMetricsReporter.scala | 4 +-
.../functional/TestPartialUpdateAvroPayload.scala | 5 +-
.../TestPartitionStatsIndexWithSql.scala | 369 +++++++++++++++++----
.../hudi/functional/TestRecordLevelIndex.scala | 4 +-
.../functional/TestRecordLevelIndexWithSQL.scala | 64 +++-
.../functional/TestSecondaryIndexPruning.scala | 1 +
.../functional/TestSixToFiveDowngradeHandler.scala | 2 +-
.../TestSparkDataSourceDAGExecution.scala | 7 +-
.../hudi/functional/TestSparkSqlCoreFlow.scala | 1 +
.../TestSparkSqlWithTimestampKeyGenerator.scala | 148 +++++++++
.../apache/hudi/functional/TestSqlStatement.scala | 1 +
.../hudi/functional/TestStreamingSource.scala | 46 ++-
.../hudi/functional/cdc/HoodieCDCTestBase.scala | 2 +-
.../functional/cdc/TestCDCDataFrameSuite.scala | 1 +
.../org/apache/hudi/util/JavaConversions.scala | 3 +-
.../scala/org/apache/hudi/util/TestPathUtils.scala | 1 -
.../util/TestSparkInternalSchemaConverter.scala | 3 +-
.../spark/hudi/benchmark/HoodieBenchmark.scala | 9 +-
.../spark/hudi/benchmark/HoodieBenchmarkBase.scala | 1 +
.../spark/hudi/benchmark/HoodieBenchmarks.scala | 4 +-
.../org/apache/spark/sql/avro/TestAvroSerDe.scala | 4 +-
.../spark/sql/avro/TestSchemaConverters.scala | 1 +
.../execution/benchmark/AvroSerDerBenchmark.scala | 3 +-
.../benchmark/BoundInMemoryExecutorBenchmark.scala | 3 +-
.../benchmark/CowTableReadBenchmark.scala | 9 +-
.../benchmark/LSMTimelineReadBenchmark.scala | 5 +-
.../ReadAndWriteWithoutAvroBenchmark.scala | 5 +-
.../benchmark/SpaceCurveOptimizeBenchmark.scala | 3 +-
.../TestHoodiePruneFileSourcePartitions.scala | 1 +
.../hudi/command/index/TestFunctionalIndex.scala | 5 +
.../sql/hudi/command/index/TestIndexSyntax.scala | 1 +
.../hudi/command/index/TestSecondaryIndex.scala | 1 +
.../sql/hudi/common/HoodieSparkSqlTestBase.scala | 17 +-
.../hudi/common/TestHoodieInternalRowUtils.scala | 5 +-
.../sql/hudi/common/TestHoodieOptionConfig.scala | 1 +
.../TestNestedSchemaPruningOptimization.scala | 1 +
.../apache/spark/sql/hudi/common/TestSqlConf.scala | 1 +
.../sql/hudi/ddl/TestAlterTableDropPartition.scala | 41 ++-
.../spark/sql/hudi/dml/TestDataSkippingQuery.scala | 6 +
.../spark/sql/hudi/dml/TestDeleteTable.scala | 1 +
.../apache/spark/sql/hudi/dml/TestDropTable.scala | 3 +-
.../hudi/dml/TestHoodieTableValuedFunction.scala | 1 +
.../spark/sql/hudi/dml/TestInsertTable.scala | 104 +++++-
.../sql/hudi/dml/TestMergeIntoLogOnlyTable.scala | 1 +
.../spark/sql/hudi/dml/TestMergeIntoTable.scala | 1 +
.../TestMergeIntoTableWithNonRecordKeyField.scala | 1 +
.../hudi/dml/TestPartialUpdateForMergeInto.scala | 2 +-
.../spark/sql/hudi/dml/TestRepairTable.scala | 1 +
.../spark/sql/hudi/dml/TestShowPartitions.scala | 1 +
.../spark/sql/hudi/dml/TestTruncateTable.scala | 1 +
.../sql/hudi/procedure/TestCallCommandParser.scala | 2 +
.../hudi/procedure/TestClusteringProcedure.scala | 4 +-
.../hudi/procedure/TestCompactionProcedure.scala | 3 +-
.../procedure/TestCopyToTempViewProcedure.scala | 1 +
...tRunRollbackInflightTableServiceProcedure.scala | 122 +++++++
.../procedure/TestShowFileStatusProcedure.scala | 330 ++++++++++++++++++
.../TestShowInvalidParquetProcedure.scala | 1 -
.../TestUpgradeOrDowngradeProcedure.scala | 2 +-
.../apache/spark/sql/adapter/Spark2Adapter.scala | 7 +
.../apache/hudi/spark3/internal/DefaultSource.java | 1 +
.../spark3/internal/HoodieWriterCommitMessage.java | 1 +
.../spark/sql/HoodieSpark3CatalystPlanUtils.scala | 4 +-
.../spark/sql/adapter/BaseSpark3Adapter.scala | 16 +-
.../Spark3ParquetSchemaEvolutionUtils.scala | 9 +-
.../datasources/Spark3ParsePartitionUtil.scala | 6 +-
.../spark/sql/HoodieSpark33CatalystPlanUtils.scala | 3 +-
.../apache/spark/sql/adapter/Spark3_3Adapter.scala | 5 +-
.../apache/spark/sql/avro/AvroDeserializer.scala | 1 +
.../org/apache/spark/sql/avro/AvroSerializer.scala | 11 +-
.../org/apache/spark/sql/avro/AvroUtils.scala | 11 +-
.../datasources/Spark33NestedSchemaPruning.scala | 1 +
.../Spark33LegacyHoodieParquetFileFormat.scala | 1 +
.../hudi/Spark33ResolveHudiAlterTableCommand.scala | 1 +
.../sql/hudi/analysis/HoodieSpark33Analysis.scala | 2 +-
.../HoodieSpark3_3ExtendedSqlAstBuilder.scala | 6 +-
.../parser/HoodieSpark3_3ExtendedSqlParser.scala | 5 +-
.../java/org/apache/hudi/hive/HiveSyncConfig.java | 8 +
.../java/org/apache/hudi/hive/HiveSyncTool.java | 101 ++++--
.../org/apache/hudi/hive/HoodieHiveSyncClient.java | 50 +++
.../hudi/hive/MultiPartKeysValueExtractor.java | 2 +-
.../hudi/hive/transaction/lock/Heartbeat.java | 3 +-
.../apache/hudi/hive/util/FilterGenVisitor.java | 2 +-
.../hudi/hive/util/PartitionFilterGenerator.java | 8 +-
.../org/apache/hudi/hive/TestHiveSyncTool.java | 333 +++++++++++++++++++
.../hudi/hive/TestMultiPartKeysValueExtractor.java | 3 +-
.../hudi/hive/TestParquet2SparkSchemaUtils.java | 3 +-
.../apache/hudi/hive/testutils/HiveTestUtil.java | 130 +++++++-
.../hive/util/TestPartitionFilterGenerator.java | 1 +
.../hudi/sync/common/HoodieMetaSyncOperations.java | 29 ++
.../apache/hudi/sync/common/HoodieSyncClient.java | 10 +-
.../apache/hudi/sync/common/HoodieSyncConfig.java | 12 +
.../apache/hudi/sync/common/HoodieSyncTool.java | 3 +
.../sync/common/metrics/HoodieMetaSyncMetrics.java | 118 +++++++
.../sync/common/util/Parquet2SparkSchemaUtils.java | 1 +
.../common/metrics/TestHoodieMetaSyncMetrics.java | 106 ++++++
.../service/handlers/marker/MarkerDirState.java | 2 +-
.../org/apache/hudi/utilities/HoodieCleaner.java | 2 +-
.../hudi/utilities/HoodieDataTableValidator.java | 3 +-
.../utilities/HoodieMetadataTableValidator.java | 24 +-
.../apache/hudi/utilities/HoodieRepairTool.java | 2 +-
.../hudi/utilities/HoodieSnapshotExporter.java | 60 +++-
.../apache/hudi/utilities/schema/SchemaSet.java | 4 +-
.../utilities/schema/SimpleSchemaProvider.java | 3 +-
.../postprocessor/ChainedSchemaPostProcessor.java | 2 +-
.../hudi/utilities/sources/ORCDFSSource.java | 1 +
.../hudi/utilities/sources/S3EventsSource.java | 4 +-
.../sources/SnapshotLoadQuerySplitter.java | 1 +
.../utilities/sources/helpers/AvroConvertor.java | 2 +-
.../utilities/sources/helpers/QueryRunner.java | 2 +-
.../sources/helpers/S3EventsMetaSelector.java | 4 +-
.../transform/SqlFileBasedTransformer.java | 9 +-
.../transform/SqlQueryBasedTransformer.java | 8 +-
.../apache/hudi/utilities/TestHoodieIndexer.java | 6 +-
.../TestHoodieMetadataTableValidator.java | 13 +-
.../callback/TestKafkaCallbackProvider.java | 1 +
.../deltastreamer/TestHoodieDeltaStreamer.java | 15 +-
.../TestErrorTableAwareChainedTransformer.java | 6 +-
.../functional/TestHoodieSnapshotExporter.java | 133 +++++++-
.../TestHoodieMultiTableServicesMain.java | 9 +-
.../hudi/utilities/sources/TestAvroDFSSource.java | 2 +
.../hudi/utilities/sources/TestS3EventsSource.java | 2 +-
.../helpers/TestSnapshotQuerySplitterImpl.java | 2 +
...ClientFunctionalTestHarnessWithHiveSupport.java | 1 +
.../transform/TestSqlQueryBasedTransformer.java | 5 +
.../sql-file-transformer.sql | 10 +-
pom.xml | 8 +-
release/release_guide.md | 6 +-
rfc/README.md | 3 +-
scripts/release/deploy_staging_jars.sh | 3 +-
scripts/release/validate_staged_bundles.sh | 2 +-
670 files changed, 6120 insertions(+), 1607 deletions(-)
copy
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/multisync/MockSyncTool2.java
=>
hudi-aws/src/test/java/org/apache/hudi/aws/sync/MockAwsGlueCatalogSyncTool.java
(65%)
create mode 100644
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAWSGlueSyncClient.java
create mode 100644
hudi-aws/src/test/java/org/apache/hudi/aws/sync/TestAwsGlueSyncTool.java
create mode 100644
hudi-aws/src/test/java/org/apache/hudi/aws/testutils/GlueTestUtil.java
copy hudi-common/src/test/resources/{simple-test-evolved-compatible.avsc =>
complex-schema-evolved.avsc} (83%)
create mode 100644 hudi-common/src/test/resources/complex-schema-evolved.data
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/RunRollbackInflightTableServiceProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowFileStatusProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestParquetReaderCompatibility.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithTimestampKeyGenerator.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestRunRollbackInflightTableServiceProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestShowFileStatusProcedure.scala
create mode 100644
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/metrics/HoodieMetaSyncMetrics.java
create mode 100644
hudi-sync/hudi-sync-common/src/test/java/org/apache/hudi/sync/common/metrics/TestHoodieMetaSyncMetrics.java
copy hudi-utilities/src/test/resources/{streamer-config =>
exporter}/sql-file-transformer.sql (88%)
