Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 On Tue, Apr 16, 2024 at 12:50 AM Josh Rosen wrote: > +1 > > On Mon, Apr 15, 2024 at 11:26 AM Maciej wrote: > >> +1 >> >> Best regards, >> Maciej Szymkiewicz >> >> Web: https://zero323.net >> PGP: A30CEF0C31A501EC >> >> On 4/15/24 8:16 PM, Rui Wang wrote: >> >> +1, non-binding. >> >> Thanks

Re: [VOTE] Release Spark 3.4.3 (RC2)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pkubernetes Regards, Mridul On Sun, Apr 14, 2024 at 11:31 PM Dongjoon Hyun wrote: > I'll start with my +1. > > - Checked checksum and signature > - Checked Scala/Java/R/Python/SQL Document's

[VOTE][RESULT] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

Hi all, The vote passes with 7+1s (5 binding +1s). (* = binding) +1: Dongjoon Hyun(*) Liang-Chi Hsieh(*) Huaxin Gao(*) Bo Yang Xiao Li(*) Chao Sun(*) Hussein Awala +0: None -1: None Thanks. - To unsubscribe e-mail:

Re: Which version of spark version supports parquet version 2 ?

Any specific reason spark does not support or community doesn't want to go to Parquet V2 , which is more optimized and read and write is too much faster (form other component which I am using) On Mon, Apr 15, 2024 at 7:55 PM Ryan Blue wrote: > Spark will read data written with v2 encodings just

Request Review for [SPARK-46992]Fix cache consistency

Hi community, I've fixed the issue about consistency of cache: SPARK-46992 long time ago. I'll appreciate if someone could help review this pr! -- Best, Jay

Re: Which version of spark version supports parquet version 2 ?

Spark will read data written with v2 encodings just fine. You just don't need to worry about making Spark produce v2. And you should probably also not produce v2 encodings from other systems. On Mon, Apr 15, 2024 at 4:37 PM Prem Sahoo wrote: > oops but so spark does not support parquet V2 atm

Re: Which version of spark version supports parquet version 2 ?

oops but so spark does not support parquet V2 atm ?, as We have a use case where we need parquet V2 as one of our components uses Parquet V2 . On Mon, Apr 15, 2024 at 7:09 PM Ryan Blue wrote: > Hi Prem, > > Parquet v1 is the default because v2 has not been finalized and adopted by > the

Re: Which version of spark version supports parquet version 2 ?

Hi Prem, Parquet v1 is the default because v2 has not been finalized and adopted by the community. I highly recommend not using v2 encodings at this time. Ryan On Mon, Apr 15, 2024 at 3:05 PM Prem Sahoo wrote: > I am using spark 3.2.0 . but my spark package comes with parquet-mr 1.2.1 > which

Re: Which version of spark version supports parquet version 2 ?

I am using spark 3.2.0 . but my spark package comes with parquet-mr 1.2.1 which writes in parquet version 1 not version version 2:(. so I was looking how to write in Parquet version2 ? On Mon, Apr 15, 2024 at 5:05 PM Mich Talebzadeh wrote: > Sorry you have a point there. It was released in

Re: Which version of spark version supports parquet version 2 ?

Sorry you have a point there. It was released in version 3.00. What version of spark are you using? Technologist | Solutions Architect | Data Engineer | Generative AI London United Kingdom view my Linkedin profile

Re: Which version of spark version supports parquet version 2 ?

Thank you so much for the info! But do we have any release notes where it says spark2.4.0 onwards supports parquet version 2. I was under the impression Spark3.0 onwards it started supporting . On Mon, Apr 15, 2024 at 4:28 PM Mich Talebzadeh wrote: > Well if I am correct, Parquet version 2

Re: Which version of spark version supports parquet version 2 ?

Well if I am correct, Parquet version 2 support was introduced in Spark version 2.4.0. Therefore, any version of Spark starting from 2.4.0 supports Parquet version 2. Assuming that you are using Spark version 2.4.0 or later, you should be able to take advantage of Parquet version 2 features. HTH

Re: Which version of spark version supports parquet version 2 ?

Thank you for the information! I can use any version of parquet-mr to produce parquet file. regarding 2nd question . Which version of spark is supporting parquet version 2? May I get the release notes where parquet versions are mentioned ? On Mon, Apr 15, 2024 at 2:34 PM Mich Talebzadeh wrote:

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 On Mon, Apr 15, 2024 at 11:26 AM Maciej wrote: > +1 > > Best regards, > Maciej Szymkiewicz > > Web: https://zero323.net > PGP: A30CEF0C31A501EC > > On 4/15/24 8:16 PM, Rui Wang wrote: > > +1, non-binding. > > Thanks Dongjoon to drive this! > > > -Rui > > On Mon, Apr 15, 2024 at 10:10 AM

Re: Which version of spark version supports parquet version 2 ?

Parquet-mr is a Java library that provides functionality for working with Parquet files with hadoop. It is therefore more geared towards working with Parquet files within the Hadoop ecosystem, particularly using MapReduce jobs. There is no definitive way to check exact compatible versions within

Re: Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 On Mon, Apr 15, 2024 at 11:17 AM Rui Wang wrote: > +1, non-binding. > > Thanks Dongjoon to drive this! > > > -Rui > > On Mon, Apr 15, 2024 at 10:10 AM Xinrong Meng wrote: > >> +1 >> >> Thank you @Dongjoon Hyun ! >> >> On Mon, Apr 15, 2024 at 6:33 AM beliefer wrote: >> >>> +1 >>> >>> >>> 在

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 Best regards, Maciej Szymkiewicz Web:https://zero323.net PGP: A30CEF0C31A501EC On 4/15/24 8:16 PM, Rui Wang wrote: +1, non-binding. Thanks Dongjoon to drive this! -Rui On Mon, Apr 15, 2024 at 10:10 AM Xinrong Meng wrote: +1 Thank you @Dongjoon Hyun

Re: Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1, non-binding. Thanks Dongjoon to drive this! -Rui On Mon, Apr 15, 2024 at 10:10 AM Xinrong Meng wrote: > +1 > > Thank you @Dongjoon Hyun ! > > On Mon, Apr 15, 2024 at 6:33 AM beliefer wrote: > >> +1 >> >> >> 在 2024-04-15 15:54:07，"Peter Toth" 写道： >> >> +1 >> >> Wenchen Fan ezt írta

Re: Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 Thank you @Dongjoon Hyun ! On Mon, Apr 15, 2024 at 6:33 AM beliefer wrote: > +1 > > > 在 2024-04-15 15:54:07，"Peter Toth" 写道： > > +1 > > Wenchen Fan ezt írta (időpont: 2024. ápr. 15., H, > 9:08): > >> +1 >> >> On Sun, Apr 14, 2024 at 6:28 AM Dongjoon Hyun >> wrote: >> >>> I'll start from

Which version of spark version supports parquet version 2 ?

Hello Team, May I know how to check which version of parquet is supported by parquet-mr 1.2.1 ? Which version of parquet-mr is supporting parquet version 2 (V2) ? Which version of spark is supporting parquet version 2? May I get the release notes where parquet versions are mentioned ?

Re:Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 在 2024-04-15 15:54:07，"Peter Toth" 写道： +1 Wenchen Fan ezt írta (időpont: 2024. ápr. 15., H, 9:08): +1 On Sun, Apr 14, 2024 at 6:28 AM Dongjoon Hyun wrote: I'll start from my +1. Dongjoon. On 2024/04/13 22:22:05 Dongjoon Hyun wrote: > Please vote on SPARK-4 to use ANSI SQL

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 Wenchen Fan ezt írta (időpont: 2024. ápr. 15., H, 9:08): > +1 > > On Sun, Apr 14, 2024 at 6:28 AM Dongjoon Hyun wrote: > >> I'll start from my +1. >> >> Dongjoon. >> >> On 2024/04/13 22:22:05 Dongjoon Hyun wrote: >> > Please vote on SPARK-4 to use ANSI SQL mode by default. >> > The

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1, non-binding Thanks, Cheng Pan > On Apr 15, 2024, at 14:14, John Zhuge wrote: > > +1 (non-binding) > > On Sun, Apr 14, 2024 at 7:18 PM Jungtaek Lim > wrote: > +1 (non-binding), thanks Dongjoon. > > On Sun, Apr 14, 2024 at 7:22 AM Dongjoon Hyun wrote: > Please vote on SPARK-4 to

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 (non-binding) On Sun, Apr 14, 2024 at 7:18 PM Jungtaek Lim wrote: > +1 (non-binding), thanks Dongjoon. > > On Sun, Apr 14, 2024 at 7:22 AM Dongjoon Hyun > wrote: > >> Please vote on SPARK-4 to use ANSI SQL mode by default. >> The technical scope is defined in the following PR which is