Thank you so much for the info! But do we have any release notes where it says spark2.4.0 onwards supports parquet version 2. I was under the impression Spark3.0 onwards it started supporting .
On Mon, Apr 15, 2024 at 4:28 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Well if I am correct, Parquet version 2 support was introduced in Spark > version 2.4.0. Therefore, any version of Spark starting from 2.4.0 supports > Parquet version 2. Assuming that you are using Spark version 2.4.0 or > later, you should be able to take advantage of Parquet version 2 features. > > HTH > > Mich Talebzadeh, > Technologist | Solutions Architect | Data Engineer | Generative AI > London > United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* The information provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > On Mon, 15 Apr 2024 at 20:53, Prem Sahoo <prem.re...@gmail.com> wrote: > >> Thank you for the information! >> I can use any version of parquet-mr to produce parquet file. >> >> regarding 2nd question . >> Which version of spark is supporting parquet version 2? >> May I get the release notes where parquet versions are mentioned ? >> >> >> On Mon, Apr 15, 2024 at 2:34 PM Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> Parquet-mr is a Java library that provides functionality for working >>> with Parquet files with hadoop. It is therefore more geared towards >>> working with Parquet files within the Hadoop ecosystem, particularly using >>> MapReduce jobs. There is no definitive way to check exact compatible >>> versions within the library itself. However, you can have a look at this >>> >>> https://github.com/apache/parquet-mr/blob/master/CHANGES.md >>> >>> HTH >>> >>> Mich Talebzadeh, >>> Technologist | Solutions Architect | Data Engineer | Generative AI >>> London >>> United Kingdom >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* The information provided is correct to the best of my >>> knowledge but of course cannot be guaranteed . It is essential to note >>> that, as with any advice, quote "one test result is worth one-thousand >>> expert opinions (Werner >>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>> >>> >>> On Mon, 15 Apr 2024 at 18:59, Prem Sahoo <prem.re...@gmail.com> wrote: >>> >>>> Hello Team, >>>> May I know how to check which version of parquet is supported by >>>> parquet-mr 1.2.1 ? >>>> >>>> Which version of parquet-mr is supporting parquet version 2 (V2) ? >>>> >>>> Which version of spark is supporting parquet version 2? >>>> May I get the release notes where parquet versions are mentioned ? >>>> >>>
