This is an automated email from the ASF dual-hosted git repository. forwardxu pushed a change to branch release-0.12.1 in repository https://gitbox.apache.org/repos/asf/hudi.git
from 2a3b0b5af8a add DeleteFsFileProcedure new d8043d48ea0 fix metatable hbase-site new ebf99a7e84f [HUDI-5278] Support more conf to cluster procedure (#7304) new 9a9fd985f2f improve BackupInvalidParquetProcedure new 0344765bdf3 [HUDI-5671] BucketIndexPartitioner partition algorithm skew (#7815) new 3c4a403a0a3 [HUDI-5318] Fix partition pruning for clustering scheduling (#7366) new b0a0912cfaf [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy (#7372) new 2c8586f5a3b fix enable metrics on new 5e2db20bb20 fix Zhiyan metrics reporter new cca17a599d6 [MINOR] improve RunClusteringProcedure with partition selected new 59094436c9d [HUDI-5343] HoodieFlinkStreamer supports async clustering for append mode (#7403) new 09df7305557 [HUDI-5515] Fix concurrency conflict in ClusteringOperator with latency marker (#7625) new 94ee9a133ad [HUDI-5543] Description of clustering.plan.partition.filter.mode supports DAY_ROLLING strategy (#7656) new 700717c7344 [HUDI-5235] Clustering target size should larger than small file limit (#7232) new ee779fe86fe [HUDI-5341] CleanPlanner retains earliest commits must not be later than earliest pending commit (#7568) new e44285447a1 improve getCommitInstantsToArchive data duplication new c265ddcc4b5 [HUDI-5506] StreamWriteOperatorCoordinator may not recommit with partial uncommitted write metadata event (#7611) new 108c6afd308 [HUDI-4968] Update misleading read.streaming.skip_compaction/skip_clustering config (#6856) new 0f056e52e58 [HUDI-5286] UnsupportedOperationException throws when enabling filesystem retry (#7313) new 7588b918de2 add DropPartitionsProcedure new 45bef56db55 [HUDI-5495] add some property to table config The 20 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .idea/vcs.xml | 1 - .../apache/hudi/client/HoodieTimelineArchiver.java | 54 ++-- .../apache/hudi/config/HoodieClusteringConfig.java | 56 +++- .../hudi/table/action/clean/CleanPlanner.java | 19 +- .../PartitionAwareClusteringPlanStrategy.java | 26 +- .../TestPartitionAwareClusteringPlanStrategy.java | 2 +- .../SparkSizeBasedClusteringPlanStrategy.java | 14 +- .../SparkSortAndSizeExecutionStrategy.java | 6 +- .../TestSparkSizeBasedClusteringPlanStrategy.java | 94 ++++++ .../common/fs/HoodieRetryWrapperFileSystem.java | 5 + .../hudi/common/table/HoodieTableMetaClient.java | 19 ++ .../apache/hudi/common/util/ClusteringUtils.java | 37 +++ hudi-common/src/main/resources/hbase-site.xml | 2 +- .../fs/TestFSUtilsWithRetryWrapperEnable.java | 14 + .../hudi/common/util/TestClusteringUtils.java | 69 +++++ .../apache/hudi/configuration/FlinkOptions.java | 21 +- .../hudi/sink/StreamWriteOperatorCoordinator.java | 8 +- .../sink/bucket/BucketStreamWriteFunction.java | 5 +- .../hudi/sink/clustering/ClusteringOperator.java | 63 ++-- .../sink/clustering/FlinkClusteringConfig.java | 37 ++- .../hudi/sink/compact/FlinkCompactionConfig.java | 30 +- .../sink/partitioner/BucketIndexPartitioner.java | 5 +- .../apache/hudi/streamer/FlinkStreamerConfig.java | 53 +++- .../apache/hudi/streamer/HoodieFlinkStreamer.java | 21 +- .../java/org/apache/hudi/util/StreamerUtil.java | 57 ++-- .../sink/TestStreamWriteOperatorCoordinator.java | 16 + .../org/apache/hudi/HoodieSparkSqlWriter.scala | 4 +- .../procedures/BackupInvalidParquetProcedure.scala | 23 +- .../procedures/DropPartitionsProcedure.scala | 118 +++++++ .../hudi/command/procedures/HoodieProcedures.scala | 1 + .../procedures/RunClusteringProcedure.scala | 168 +++++++--- .../hudi/functional/TestLayoutOptimization.scala | 3 +- .../TestBackupInvalidParquetProcedure.scala | 19 +- .../hudi/procedure/TestClusteringProcedure.scala | 339 ++++++++++++++++++++- .../procedure/TestDropPartitionsProcedure.scala | 67 ++++ 35 files changed, 1263 insertions(+), 213 deletions(-) create mode 100644 hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/clustering/plan/strategy/TestSparkSizeBasedClusteringPlanStrategy.java create mode 100644 hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/DropPartitionsProcedure.scala create mode 100644 hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestDropPartitionsProcedure.scala