Sounds good.. Also the avro versions? If we can make a profile around this, we could give it a shot..
I reckon most of the world will be moving towards Spark 3, soon though (say end of 2020) On Sat, Mar 21, 2020 at 12:19 PM Pratyaksh Sharma <pratyaks...@gmail.com> wrote: > Hi Vinoth, > > It was a while ago but I guess the error was related to parquet-avro > mismatch only. Let me try reproducing it and will file a lira this time. > > On Tue, Feb 25, 2020 at 12:29 PM Vinoth Chandar <vin...@apache.org> wrote: > > > Hi Pratyaksh, > > > > Actually we voted on the mailing list here, to move over to 2.4.. :) > With > > 3.0 on the horizon, it may make sense to actually upgrade? > > Can you file an issue with how exactly 0.5.1 is failing with 2.3? are you > > hitting issues with parquet-avro mismatches? > > I am wondering if we can do a custom build overriding few versions of > > parquet/avro... (spark-avro 2.4.4 should interplay with spark 2.3?) > > > > Thanks > > VInoth > > > > On Sun, Feb 23, 2020 at 2:02 PM Pratyaksh Sharma <pratyaks...@gmail.com> > > wrote: > > > > > Hi, > > > > > > As discussed in last to last week's weekly sync, I want to put forward > > this > > > point on our mailing list also. Since with 0.5.1 release, we have > > upgraded > > > spark to 2.4 in our master branch, we are facing difficulties after > > > rebasing our codebase with master. At our organisation we are using > spark > > > 2.3.2 cluster in production and spark 2.3 to 2.4 upgrade has breaking > > > changes. > > > > > > So I wanted to know if any workaround is possible to bring back support > > for > > > spark 2.3? I am sure there are other guys/organisations using Hudi who > > are > > > not yet on spark 2.4. > > > > > > We really do not want to miss out on any of the cool features being > > > developed by the community. > > > > > >
