Hi Vinoth, > I wonder if it's possible to structure the code in separate modules, but have a single bundle
Yes, this is possible, initially I started doing the same in this PR - https://github.com/apache/hudi/pull/3162 , hence wanted to discuss this here, if we create a single bundle, we have to make sure there are no dependency conflicts. For example: Flink-Hudi is using a different version of parquet as compared to Spark-Hudi because of which the tests started to fail with `java.lang.NoSuchMethodError: org.apache.parquet.column.ParquetProperties.getColumnIndexTruncateLength()` Regards, Vinay Patil On Tue, Jul 20, 2021 at 9:46 PM Vinoth Chandar <[email protected]> wrote: > Hi Vinay. > > Thanks for kicking this off. > > I wonder if it's possible to structure the code in separate modules, but > have a single bundle. > Or is that a painful experience? (if so, can you share what issues we are > running into?) > > We have rarely done backwards incompatible changes and users appreciate > that. > So love to understand why this is warranted here. > > Thanks > Vinoth > > On Sat, Jul 17, 2021 at 7:58 AM Vinay Patil <[email protected]> > wrote: > > > Hi Team, > > > > As part of https://issues.apache.org/jira/browse/HUDI-1872, we are > > creating > > a separate flink-utilities module. Based on our discussion on the PR, > > should we even create a spark-utilities module. This would look like : > > > > hudi-utilities > > ├── hudi-flink-utilities > > └── hudi-spark-utilities > > > > This would also mean to create separate utilities-bundle for Flink and > > Spark, > > > > hudi-utilities-bundle > > ├── hudi-flink-utilities-bundle > > └── hudi-spark-utilities-bundle > > > > This is not a backward compatible change as users will have to provide an > > engine specific bundle. IMO, since Hudi is supporting Flink and Spark it > > will be good to have engine specific bundle. > > > > What do you think? > > > > Regards, > > Vinay Patil > > >
