Dear community, Nice to share Hudi community bi-weekly updates for 2020-12-06 ~ 2020-12-20 with updates on features and bug fixes.
======================================= Discussion [Best Practise] There is a discussion about Hudi Record Key Best Practices, you would check the discussion if you have some questions. [1] [CI/DI] A discussion about speeding up CI/CD build to the PRs, please chime in if you have some new ideas. [2] [Core] A discussion about supporting parallel writing to Hudi tables, this resolves some of the outstanding requirements [3] [Core] Time Travel (querying the historical versions of data) ability for Hudi Table [4] [Community] There is a proposal about organising one event to list the accomplishments and roadmap of the community [5] [Release] There is a discussion about 0.7.0 release planning, which would like release by Dec 31 [6] [Core] There is a discussion about SQL Support using Apache Calcite, which makes writing data to hudi via sql [7] ======================================= Features [Config] Make HoodieWriteConfig support setting different default value according to engine type [8] [Spark Integration] Make Hudi support Spark 3 [9] [Core] Refactor AbstractHoodieLogRecordScanner to use Builder [10] [Meta Sync] Hudi dla sync support skip rt table syncing [11] [Spark Integration] Drop Hudi metadata cols at the beginning of Spark datasource writing [12] [DeltaStreamer] Add date partition based source input selector for Deltastreamer [13] [Core] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue [14] [Core] Add base implementation for hudi java client [15] ======================================= Bugs [Writer Core] Fix partition path using FSUtils [16] [Spark Integration] Remove scala dependency from hudi-client-common [17] [Writer Core] Clean old fileslice is invalid [18] [Index] Fix bug in Marker File Reconciliation for Non-Partitioned datasets [19] [Spark Integration] support more accurate spark JobGroup for better performance tracking [20] [Integration Test] Use the latest writer schema, when reading from existing parquet files in the hudi-test-suite [21] [1] https://lists.apache.org/thread.html/r27792b6d0b354c7b6bbb7a258cdd7af14cbe3fdd777137fd619e9f63%40%3Cdev.hudi.apache.org%3E [2] https://lists.apache.org/thread.html/r1e69b6dac9b2d27a3f7c06491ac16dc5c0b5bd8e0807f4d9782b8e77%40%3Cdev.hudi.apache.org%3E [3] https://lists.apache.org/thread.html/r412c97452218f461e9bb52bc4a2f795609ec8eec5b3da1a60b9aa050%40%3Cdev.hudi.apache.org%3E [4] https://lists.apache.org/thread.html/rf978b608a5ebc3d7580b004da1a53f06ac3aaa2bb91ad069adc869f3%40%3Cdev.hudi.apache.org%3E [5] https://lists.apache.org/thread.html/r1d1b414c01cba2f127ab5e5b9aca314464ed433e11eae43b25d7c65a%40%3Cdev.hudi.apache.org%3E [6] https://lists.apache.org/thread.html/rf2ae5b4946440a0fea0e74f188db23f2099fbceaf9631bf35a4633ee%40%3Cdev.hudi.apache.org%3E [7] https://lists.apache.org/thread.html/ra04c70186f5880899ebbc8e87ed66c4b166c8e3ee062e0b8901ca6fc%40%3Cdev.hudi.apache.org%3E [8] https://issues.apache.org/jira/browse/HUDI-1412 [9] https://issues.apache.org/jira/browse/HUDI-1040 [10] https://issues.apache.org/jira/browse/HUDI-1445 [11] https://issues.apache.org/jira/browse/HUDI-1448 [12] https://issues.apache.org/jira/browse/HUDI-1376 [13] https://issues.apache.org/jira/browse/HUDI-1406 [14] https://issues.apache.org/jira/browse/HUDI-115 [15] https://issues.apache.org/jira/browse/HUDI-1419 [16] https://issues.apache.org/jira/browse/HUDI-1395 [17] https://issues.apache.org/jira/browse/HUDI-1439 [18] https://issues.apache.org/jira/browse/HUDI-1428 [19] https://issues.apache.org/jira/browse/HUDI-1435 [20] https://issues.apache.org/jira/browse/HUDI-1437 [21] https://issues.apache.org/jira/browse/HUDI-1470 Best, Leesf
