Spark3.3 in OSS built with parquet 1.12. Just compiling with parquet 1.10 results in build failure , so just wondering if any one have build & compiled Spark 3.3 with parquet 1.10.
Regards Pralabh Kumar On Mon, Jul 24, 2023 at 3:04 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi, > > Where is this limitation coming from (using 1.1.0)? That is 2018 build > > Have you tried Spark 3.3 with parquet writes as is? Just a small PoC will > prove it. > > HTH > Mich Talebzadeh, > Solutions Architect/Engineering Lead > Palantir Technologies Limited > London > United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Mon, 24 Jul 2023 at 10:15, Pralabh Kumar <pralabhku...@gmail.com> > wrote: > >> Hi Dev community. >> >> I have a quick question with respect to Spark 3.3. Currently Spark 3.3 is >> built with parquet 1.12. >> However, anyone tried Spark 3.3 with parquet 1.10 . >> >> We are at Uber , planning to migrate Spark 3.3 but we have limitations of >> using parquet 1.10 . Has anyone tried building Spark 3.3 with parquet 1.10 >> ? What are the dos/ don't for it ? >> >> >> Regards >> Pralabh Kumar >> >
