Re: [Action required] Default Spark profile changed to 3.2

2023-06-02 Thread Y Ethan Guo
There is a hotfix (https://github.com/apache/hudi/pull/8822) merged recently to fix the default build. SPARK_HOME does not matter and the latest master builds on my end with Spark 3.3 as SPARK_HOME. On Fri, Jun 2, 2023 at 1:28 AM Vinoth Chandar wrote: > Hi, > > Just tried doing a mvn clean

Re: [Action required] Default Spark profile changed to 3.2

2023-06-02 Thread Vinoth Chandar
Hi, Just tried doing a mvn clean install -DskipTests, and the build failed. My local SPARK_HOME is pointing to spark 3.3 installation. Does that all matter now? Quite possible this is an issue with my setup, just flagging. Thanks Vinoth On Fri, May 26, 2023 at 8:30 AM Shiyan Xu wrote: > Hi

Re: [DISCUSSION] Simplify code structure for supporting multiple Spark versions in Hudi

2023-06-02 Thread Y Ethan Guo
Hey Shawn, Rahil, Thanks for raising this issue. These are good suggestions; I would recommend simplifying the code structure of Hudi Spark incrementally and gradually making the code less coupled with Spark engine. Identify breaking changes introduced by the new Spark version and patch >

Re: [DISCUSSION] Simplify code structure for supporting multiple Spark versions in Hudi

2023-06-02 Thread Vinoth Chandar
This is a good topic, thanks for raising this. Overall our reliance on spark classes/APIs that are declared experimental is an issue on paper. But there is few other ways to get right performance without relying on these. This has been the tricky issue IMO. Thoughts? I ll review the code

Re: [ANNOUNCE] Apache Hudi 0.13.1 released

2023-06-02 Thread Vinoth Chandar
Thanks for driving this! On Wed, May 31, 2023 at 10:00 Yue Zhang wrote: > The Apache Hudi team is pleased to announce the release of Apache Hudi > 0.13.1 > > Apache Hudi (pronounced Hoodie) stands for Hadoop Upserts Deletes and > Incrementals. Apache Hudi manages storage of large analytical