Dear community, Nice to share Hudi community bi-weekly updates for 2021-09-12 ~ 2021-09-26 with updates on features, bug fixes and tests.
======================================= Features [Core] Add metrics-jmx to spark and flink bundles [1] [Java Client] Adding support for merge-on-read tables [2] [Flink Integration] Incremental read for Flink [3] [Flink Integration] Consume as mini-batch for flink stream reader [4] [1] https://issues.apache.org/jira/browse/HUDI-2404 [2] https://issues.apache.org/jira/browse/HUDI-2335 [3] https://issues.apache.org/jira/browse/HUDI-2449 [4] https://issues.apache.org/jira/browse/HUDI-2485 ======================================= Bugs [Hive Integration] Add --enable-sync parameter [1] [Core] Fix getDefaultBootstrapIndexClass logical error [2] [Flink Integration] Catch the throwable when scheduling the cleaning task for flink writer [3] [Kafka Connect] Fix protocol and other issues after stress testing Hudi Kafka Connect [4] [Flink Integraion] Make decimal compatible with hudi for flink writer [5] [Core] Refactor rollback actions in hudi-client module [6] [Core] Archive service executed after cleaner finished [7] [Core] Separate some config logic from HoodieMetricsConfig into HoodieMetricsGraphiteConfig HoodieMetricsJmxConfig [8] [Core] Adding rollback plan and rollback requested instant [9] [Core] Make periodSeconds of GraphiteReporter configurable [10] [Core] Fixing delete files corner cases wrt cleaning and rollback when applying changes to metadata [11] [Spark SQL] Fix the exception for mergeInto when the primaryKey and preCombineField of source table and target table differ in case only [12] [Flink Integration] HoodieFileIndex throws NPE for FileSlice with pure log files [13] [Spark Integration] Clean the marker files after compaction [14] [Hive Integration] Fixing the closing of hms client [15] [Core] Make parquet dictionary encoding configurable [16] [Flink Integration] Infer changelog mode for flink compactor [17] [Deltastreamer] Fix hive sync mode setting in Deltastreamer [18] [Core] On windows client with hdfs server for wrong file separator [19] [1] https://issues.apache.org/jira/browse/HUDI-2397 [2] https://issues.apache.org/jira/browse/HUDI-2410 [3] https://issues.apache.org/jira/browse/HUDI-2421 [4] https://issues.apache.org/jira/browse/HUDI-2428 [5] https://issues.apache.org/jira/browse/HUDI-2430 [6] https://issues.apache.org/jira/browse/HUDI-2433 [7] https://issues.apache.org/jira/browse/HUDI-2355 [8] https://issues.apache.org/jira/browse/HUDI-2423 [9] https://issues.apache.org/jira/browse/HUDI-2422 [10] https://issues.apache.org/jira/browse/HUDI-2434 [11] https://issues.apache.org/jira/browse/HUDI-2444 [12] https://issues.apache.org/jira/browse/HUDI-2343 [13] https://issues.apache.org/jira/browse/HUDI-2479 [14] https://issues.apache.org/jira/browse/HUDI-2383 [15] https://issues.apache.org/jira/browse/HUDI-2248 [16] https://issues.apache.org/jira/browse/HUDI-2385 [17] https://issues.apache.org/jira/browse/HUDI-2483 [18] https://issues.apache.org/jira/browse/HUDI-2484 [19] https://issues.apache.org/jira/browse/HUDI-2451 ====================================== Tests [Tests] TestHoodieMultiTableDeltaStreamer CI failed due to exception [1] [Tests] Add DAG nodes for Spark SQL in integration test suite [2] [Tests] Metadata tests rewrite [3] [1] https://issues.apache.org/jira/browse/HUDI-2425 [2] https://issues.apache.org/jira/browse/HUDI-2388 [3] https://issues.apache.org/jira/browse/HUDI-2395 Best, Leesf
