Dear community, Nice to share Hudi community bi-weekly updates for 2021-10-10 ~ 2021-10-24 with updates on features, bug fixes and tests.
======================================= Features [Spark SQL] support 'drop partition' sql [1] [Flink Integration] Support merging small files for flink insert operation [2] [Core] Add HoodieData abstraction and refactor compaction actions in hudi-client module [3] [1] https://issues.apache.org/jira/browse/HUDI-2482 [2] https://issues.apache.org/jira/browse/HUDI-2578 [3] https://issues.apache.org/jira/browse/HUDI-2501 ======================================= Bugs [Flink] Fix metadata table for flink [1] [Core] Insert duplicate records when precombined is deactivated for "insert" operation [2] [Flink] AppendWriteFunction throws NPE when checkpointing without written data [3] [Core] Fixed wrong validation for metadataTableEnabled in Hoodie Table [4] [Core] Metadata table compaction trigger max delta commits [5] [Core] Fixing glob pattern to skip all hoodie meta paths [6] [Core] Fix clustering handle errors [7] [Flink Integration] Flink streaming reader misses the rolling over file handles [8] [Flink Integration] Support DefaultHoodieRecordPayload for flink [9] [Flink Integration] Tweak some default config options for flink [10] [Flink Integration] Embedded timeline server on JobManager [11] [Flink Integration] Shade javax.servlet for flink bundle jar [12] [Flink Integration] Simplify the view storage config properties [13] [Flink Integration] Shaded hive for flink bundle jar [14] [Flink Integration] Remove include-flink-sql-connector-hive profile from flink bundle [15] [Core] BitCaskDiskMap - avoiding hostname resolution when logging messages [16] [Flink Integration] Strength flink compaction rollback strategy [17] [Core] Replace json based payload with protobuf for Transaction protocol [18] [CI] Generate more dependency list file for other bundles [19] [Core] Metadata table compaction trigger max delta commits [20] [Core] Fix write empty array when write.precombine.field is decimal type [21] [Core] Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests [22] [Core] Metadata table support for rolling back the first commit [23] [1] https://issues.apache.org/jira/browse/HUDI-2537 [2] https://issues.apache.org/jira/browse/HUDI-2496 [3] https://issues.apache.org/jira/browse/HUDI-2542 [4] https://issues.apache.org/jira/browse/HUDI-2540 [5] https://issues.apache.org/jira/browse/HUDI-2532 [6] https://issues.apache.org/jira/browse/HUDI-2494 [7] https://issues.apache.org/jira/browse/HUDI-2435 [8] https://issues.apache.org/jira/browse/HUDI-2548 [9] https://issues.apache.org/jira/browse/HUDI-2551 [10] https://issues.apache.org/jira/browse/HUDI-2556 [11] https://issues.apache.org/jira/browse/HUDI-2562 [12] https://issues.apache.org/jira/browse/HUDI-2557 [13] https://issues.apache.org/jira/browse/HUDI-2568 [14] https://issues.apache.org/jira/browse/HUDI-2569 [15] https://issues.apache.org/jira/browse/HUDI-2571 [16] https://issues.apache.org/jira/browse/HUDI-2561 [17] https://issues.apache.org/jira/browse/HUDI-2572 [18] https://issues.apache.org/jira/browse/HUDI-2469 [19] https://issues.apache.org/jira/browse/HUDI-2507 [20] https://issues.apache.org/jira/browse/HUDI-2553 [21] https://issues.apache.org/jira/browse/HUDI-2592 [22] https://issues.apache.org/jira/browse/HUDI-2489 [23] https://issues.apache.org/jira/browse/HUDI-2468 ====================================== Tests [Tests] Fixing some test failures to unblock broken CI master [1] [Tests] Fix few Cleaner tests with metadata table enabled [2] [Tests] Fix flakiness in TestHoodieDeltaStreamer [3] [Tests] Refactor TestWriteCopyOnWrite test cases [4] [1] https://issues.apache.org/jira/browse/HUDI-2552 [2] https://issues.apache.org/jira/browse/HUDI-2472 [3] https://issues.apache.org/jira/browse/HUDI-2077 [4] https://issues.apache.org/jira/browse/HUDI-2583 Best, Leesf