Hi Vinay, I am not sure why we are bundling parquet with Flink. If so, we could try and resolve that? That's the route we can first take IMO.
Our bundles don't bundle, spark, flink, hadoop, parquet. So I think a single bundle is doable. Happy to help with specific issues as they come up. On Tue, Jul 20, 2021 at 9:35 AM Vinay Patil <vinay18.pa...@gmail.com> wrote: > Hi Vinoth, > > > I wonder if it's possible to structure the code in separate modules, but > have a single bundle > > Yes, this is possible, initially I started doing the same in this PR - > https://github.com/apache/hudi/pull/3162 , hence wanted to discuss this > here, if we create a single bundle, we have to make sure there are no > dependency conflicts. For example: Flink-Hudi is using a different version > of parquet as compared to Spark-Hudi because of which the tests started to > fail with `java.lang.NoSuchMethodError: > org.apache.parquet.column.ParquetProperties.getColumnIndexTruncateLength()` > > > Regards, > Vinay Patil > > > On Tue, Jul 20, 2021 at 9:46 PM Vinoth Chandar <vin...@apache.org> wrote: > > > Hi Vinay. > > > > Thanks for kicking this off. > > > > I wonder if it's possible to structure the code in separate modules, but > > have a single bundle. > > Or is that a painful experience? (if so, can you share what issues we are > > running into?) > > > > We have rarely done backwards incompatible changes and users appreciate > > that. > > So love to understand why this is warranted here. > > > > Thanks > > Vinoth > > > > On Sat, Jul 17, 2021 at 7:58 AM Vinay Patil <vinay18.pa...@gmail.com> > > wrote: > > > > > Hi Team, > > > > > > As part of https://issues.apache.org/jira/browse/HUDI-1872, we are > > > creating > > > a separate flink-utilities module. Based on our discussion on the PR, > > > should we even create a spark-utilities module. This would look like : > > > > > > hudi-utilities > > > ├── hudi-flink-utilities > > > └── hudi-spark-utilities > > > > > > This would also mean to create separate utilities-bundle for Flink and > > > Spark, > > > > > > hudi-utilities-bundle > > > ├── hudi-flink-utilities-bundle > > > └── hudi-spark-utilities-bundle > > > > > > This is not a backward compatible change as users will have to provide > an > > > engine specific bundle. IMO, since Hudi is supporting Flink and Spark > it > > > will be good to have engine specific bundle. > > > > > > What do you think? > > > > > > Regards, > > > Vinay Patil > > > > > >
