Re: PSA, PR merges slowed down due to flaky CI
Hey folks, We chased down the flakiness and have fixed them. CI is green now. Please rebase with latest master if you have any pending patches. thanks for the patience. On Fri, Dec 17, 2021 at 10:37 AM Sivabalan wrote: > Here is the update: > Overall we have 3 flaky tests. > https://issues.apache.org/jira/browse/HUDI-3053 > > I am looking into one of them 3043, have couple of fixes, but still not > out of the woods. Will be spending more time to get to the bottom. But if > anyone has cycles, would appreciate if you can check the other two > failures. If anyone is going to triage any of these, please post it here so > that others are aware. > > > On Thu, Dec 16, 2021 at 11:45 AM Sivabalan wrote: > >> I am looking into the Deltastreamer continuous mode multiwriter test >> failure. Will update the thread once I have some tangible progress. >> >> >> On Thu, Dec 16, 2021 at 7:20 AM Vinoth Chandar wrote: >> >>> Hi all, >>> >>> We have been fighting some flakiness in the CI. I think we were able to >>> resolve the IT tests continuously failing in Azure due to memory issues. >>> There is some residual issues still. >>> >>> We would like to take sometime to resolve these before we push on with >>> the >>> PR backlog we have. Appreciate your patience. >>> >>> Folks, working on this - if you could keep this thread updated for >>> everyone, that will be great! >>> >>> Thanks >>> Vinoth >>> >> >> >> -- >> Regards, >> -Sivabalan >> > > > -- > Regards, > -Sivabalan > -- Regards, -Sivabalan
[ANNOUNCE] Hudi Community Update(2021-12-05 ~ 2021-12-19)
Dear community, Nice to share Hudi community bi-weekly updates for 2021-12-05 ~ 2021-12-19 with updates on features, bug fixes and tests. === Features [Core] Add a hudi-trino-bundle for Trino [1] [Core] Add a repair util to clean up dangling data and log files [2] [1] https://issues.apache.org/jira/browse/HUDI-2784 [2] https://issues.apache.org/jira/browse/HUDI-2906 === Bugs [Core] Fix corrupt block end position [1] [Core] for hive/presto hudi should remove the temp file which created by HoodieMergedLogRecordSanner when the query finished [2] [Core] Fixing aws lock configs to inherit from HoodieConfig [3] [Flink] Shade kryo jar for flink bundle jar [4] [Core] Fix overflow of huge log file in HoodieLogFormatWriter [5] [Core] Cache BaseDir if HudiTableNotFound Exception thrown [6] [Core] Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScaner when the query finished [7] [Core] Fixed the bug clustering jobs cannot running in parallel [8] [Core] Improve SparkUI job description for write path [9] [Core] Fixing metadata table for non-partitioned dataset [10] [Core] Make Z-index more generic Column-Stats Index [11] [Core] Make the prefix for metrics name configurable [12] [Core] Implement #close for AbstractTableFileSystemView [13] [Build] Upgrade maven plugins to be compatible with higher Java versions [14] [Core] Metadata table util to get latest file slices for reader/writers [15] [Core] Sync to HMS when deleting partitions [16] [Core] Add judgement to existed partitionPath in the catch code block [17] [Flink] Flink streaming reader 'skip_compaction' option does not work [18] [Flink] Skip the corrupt meta file for pending rollback action [19] [Flink] Add explicit write handler for flink [20] [Core] Implement #reset and #sync for metadata filesystem view [21] [Core] lean up the marker directory when finish bootstrap operation [22] [Core] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType [23] [Core] InProcess lock provider to guard single writer process with async table operations [24] [Core] Transaction manager: avoid deadlock when doing begin and end transactions [25] [1] https://issues.apache.org/jira/browse/HUDI-2900 [2] https://issues.apache.org/jira/browse/HUDI-2876 [3] https://issues.apache.org/jira/browse/HUDI-2964 [4] https://issues.apache.org/jira/browse/HUDI-2957 [5] https://issues.apache.org/jira/browse/HUDI-2665 [6] https://issues.apache.org/jira/browse/HUDI-2779 [7] https://issues.apache.org/jira/browse/HUDI-2966 [8] https://issues.apache.org/jira/browse/HUDI-2901 [9] https://issues.apache.org/jira/browse/HUDI-2849 [10] https://issues.apache.org/jira/browse/HUDI-2952 [11] https://issues.apache.org/jira/browse/HUDI-2814 [12] https://issues.apache.org/jira/browse/HUDI-2974 [13] https://issues.apache.org/jira/browse/HUDI-2984 [14] https://issues.apache.org/jira/browse/HUDI-2946 [15] https://issues.apache.org/jira/browse/HUDI-2938 [16] https://issues.apache.org/jira/browse/HUDI-2990 [17] https://issues.apache.org/jira/browse/HUDI-2994 [18] https://issues.apache.org/jira/browse/HUDI-2996 [19] https://issues.apache.org/jira/browse/HUDI-2997 [20] https://issues.apache.org/jira/browse/HUDI-3024 [21] https://issues.apache.org/jira/browse/HUDI-3015 [22] https://issues.apache.org/jira/browse/HUDI-3001 [23] https://issues.apache.org/jira/browse/HUDI-2958 [24] https://issues.apache.org/jira/browse/HUDI-2962 [25] https://issues.apache.org/jira/browse/HUDI-3029 == Tests [Tests] Add data count checks in async clustering tests [1] [Tests] Multi writer test with conflicting async table services [2] [Tests] Adding some test fixes to continuous mode multi writer tests [3] [Tests] De-coupling multi writer tests [4] [Tests] Fixing a bug in TransactionManager and FileSystemTestLock [5] [Tests] Fixing default lock configs for FileSystemBasedLock and fixing a flaky test [6] [Tests] Fix flaky testJsonKafkaSourceResetStrategy [7] [Tests] Adding tests for archival of replace commit actions [8] [1] https://issues.apache.org/jira/browse/HUDI-2936 [2] https://issues.apache.org/jira/browse/HUDI-2527 [3] https://issues.apache.org/jira/browse/HUDI-3043 [4] https://issues.apache.org/jira/browse/HUDI-3043 [5] https://issues.apache.org/jira/browse/HUDI-3064 [6] https://issues.apache.org/jira/browse/HUDI-3054 [7] https://issues.apache.org/jira/browse/HUDI-3052 [8] https://issues.apache.org/jira/browse/HUDI-2970 Best, Leesf
