Dear community, Nice to share Hudi community bi-weekly updates for 2021-05-22 ~ 2021-06-06 with updates on features, bug fixes and tests.
======================================= Features [Flink Integration] Exactly-once write for flink writer [1] [Spark Integration] Support Partition Prune For MergeOnRead Snapshot Table [2] [Flink Integration] Improve HoodieFlinkStreamer [3] [Flink Integration] Refactor BucketAssigner to make it more efficient [4] [Flink Integration] Add target io option for flink compaction [5] [1] https://issues.apache.org/jira/browse/HUDI-1923 [2] https://issues.apache.org/jira/browse/HUDI-1879 [3] https://issues.apache.org/jira/browse/HUDI-1927 [4] https://issues.apache.org/jira/browse/HUDI-1949 [5] ttps://issues.apache.org/jira/browse/HUDI-1921 ======================================= Bugs [Spark Integration] collect() call causing issues with very large upserts [1] [Flink Integration] Type mismatch when streaming read copy_on_write table using flink[2] [Core] Set archived as the default value of HOODIE_ARCHIVELOG_FOLDER_PROP_NAME [3] [Flink Integration] Close the file handles gracefully for flink write function to avoid corrupted files [4] [Core] Fix path selector listing files with the same mod date [5] [Core] Bulk insert with row writer supports mor table [6] [Flink Integration] Make embedded time line service singleton [7] [Flink Integration] Exclude file slices in pending compaction when performing small file sizing [8] [Flink Integratoin] Shade kryo-shaded jar for hudi flink bundle [9] [Flink Integration] Lose properties when hoodieWriteConfig initializtion [10] [Flink Integration] Fix hive3 meta sync for flink writer [11] [Flink Integration] Fix NPE due to not set the output type of the operator [12] [Flink Integration] Fix flink timeline service lack jetty dependency [13] [Flink Integration] only reset bucket when flush bucket success [14] [Core] Add deltacommit to ActionType [15] [Hive Integration] Fix the NPE for MOR Hive rt table query [16] [1] https://issues.apache.org/jira/browse/HUDI-1873 [2] https://issues.apache.org/jira/browse/HUDI-1919 [3] https://issues.apache.org/jira/browse/HUDI-1920 [4] https://issues.apache.org/jira/browse/HUDI-1895 [5] https://issues.apache.org/jira/browse/HUDI-1723 [6] https://issues.apache.org/jira/browse/HUDI-1922 [7] https://issues.apache.org/jira/browse/HUDI-1865 [8] https://issues.apache.org/jira/browse/HUDI-1800 [9] https://issues.apache.org/jira/browse/HUDI-1948 [10] https://issues.apache.org/jira/browse/HUDI-1943 [11] https://issues.apache.org/jira/browse/HUDI-1952 [12] https://issues.apache.org/jira/browse/HUDI-1953 [13] https://issues.apache.org/jira/browse/HUDI-1957 [14] https://issues.apache.org/jira/browse/HUDI-1917 [15] https://issues.apache.org/jira/browse/HUDI-1281 [16] https://issues.apache.org/jira/browse/HUDI-1967 ====================================== Tests [Tests] Add SqlQueryBasedTransformer unit test [1] [Tests] Add a debezium json integration test case for flink [2] [1] https://issues.apache.org/jira/browse/HUDI-1940 [2] https://issues.apache.org/jira/browse/HUDI-1961 Best, Leesf