Re: Spark 3.3 + parquet 1.10

2023-07-24 Thread Mich Talebzadeh 
personally I have not done it myself.

CCed to spark user group if some user has tried it among users.

HTH

Mich Talebzadeh,
Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Mon, 24 Jul 2023 at 10:56, Pralabh Kumar  wrote:

> Spark3.3 in OSS built with parquet 1.12.  Just compiling  with parquet
> 1.10 results in build failure , so just wondering if any one have build &
> compiled Spark 3.3 with parquet 1.10.
>
> Regards
> Pralabh Kumar
>
> On Mon, Jul 24, 2023 at 3:04 PM Mich Talebzadeh 
> wrote:
>
>> Hi,
>>
>> Where is this limitation coming from (using 1.1.0)? That is 2018 build
>>
>> Have you tried Spark 3.3 with parquet writes as is? Just a small PoC will
>> prove it.
>>
>> HTH
>> Mich Talebzadeh,
>> Solutions Architect/Engineering Lead
>> Palantir Technologies Limited
>> London
>> United Kingdom
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Mon, 24 Jul 2023 at 10:15, Pralabh Kumar 
>> wrote:
>>
>>> Hi Dev community.
>>>
>>> I have a quick question with respect to Spark 3.3. Currently Spark 3.3
>>> is built with parquet 1.12.
>>> However, anyone tried Spark 3.3 with parquet 1.10 .
>>>
>>> We are at Uber , planning to migrate Spark 3.3 but we have limitations
>>> of using parquet 1.10  . Has anyone tried building Spark 3.3 with parquet
>>> 1.10 ? What are the dos/ don't for it ?
>>>
>>>
>>> Regards
>>> Pralabh Kumar
>>>
>>

Re: Spark 3.3 + parquet 1.10

2023-07-24 Thread Pralabh Kumar 
Spark3.3 in OSS built with parquet 1.12.  Just compiling  with parquet 1.10
results in build failure , so just wondering if any one have build &
compiled Spark 3.3 with parquet 1.10.

Regards
Pralabh Kumar

On Mon, Jul 24, 2023 at 3:04 PM Mich Talebzadeh 
wrote:

> Hi,
>
> Where is this limitation coming from (using 1.1.0)? That is 2018 build
>
> Have you tried Spark 3.3 with parquet writes as is? Just a small PoC will
> prove it.
>
> HTH
> Mich Talebzadeh,
> Solutions Architect/Engineering Lead
> Palantir Technologies Limited
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 24 Jul 2023 at 10:15, Pralabh Kumar 
> wrote:
>
>> Hi Dev community.
>>
>> I have a quick question with respect to Spark 3.3. Currently Spark 3.3 is
>> built with parquet 1.12.
>> However, anyone tried Spark 3.3 with parquet 1.10 .
>>
>> We are at Uber , planning to migrate Spark 3.3 but we have limitations of
>> using parquet 1.10  . Has anyone tried building Spark 3.3 with parquet 1.10
>> ? What are the dos/ don't for it ?
>>
>>
>> Regards
>> Pralabh Kumar
>>
>

Re: Spark 3.3 + parquet 1.10

2023-07-24 Thread Mich Talebzadeh 
Hi,

Where is this limitation coming from (using 1.1.0)? That is 2018 build

Have you tried Spark 3.3 with parquet writes as is? Just a small PoC will
prove it.

HTH
Mich Talebzadeh,
Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Mon, 24 Jul 2023 at 10:15, Pralabh Kumar  wrote:

> Hi Dev community.
>
> I have a quick question with respect to Spark 3.3. Currently Spark 3.3 is
> built with parquet 1.12.
> However, anyone tried Spark 3.3 with parquet 1.10 .
>
> We are at Uber , planning to migrate Spark 3.3 but we have limitations of
> using parquet 1.10  . Has anyone tried building Spark 3.3 with parquet 1.10
> ? What are the dos/ don't for it ?
>
>
> Regards
> Pralabh Kumar
>

Spark 3.3 + parquet 1.10

2023-07-24 Thread Pralabh Kumar 
Hi Dev community.

I have a quick question with respect to Spark 3.3. Currently Spark 3.3 is
built with parquet 1.12.
However, anyone tried Spark 3.3 with parquet 1.10 .

We are at Uber , planning to migrate Spark 3.3 but we have limitations of
using parquet 1.10  . Has anyone tried building Spark 3.3 with parquet 1.10
? What are the dos/ don't for it ?


Regards
Pralabh Kumar

Re: Spark 3.0.0 EOL

2023-07-24 Thread Dongjoon Hyun 
As Hyukjin replied, Apache Spark 3.0.0 is already in EOL status.

To Pralabh, FYI, in the community,

- Apache Spark 3.2 also reached the EOL already.
  https://lists.apache.org/thread/n4mdfwr5ksgpmrz0jpqp335qpvormos1

If you are considering Apache Spark 4, here is the other 3.x timeline,

- Apache Spark 3.3 => December, 2023.
- Apache Spark 3.4 => October, 2024
- Upcoming Apache Spark 3.5 => 18 months from the release

Thanks,
Dongjoon.


On Mon, Jul 24, 2023 at 12:21 AM Hyukjin Kwon  wrote:

>
> It's already EOL
>
> On Mon, Jul 24, 2023 at 4:17 PM Pralabh Kumar 
> wrote:
>
>> Hi Dev Team
>>
>> If possible , can you please provide the Spark 3.0.0 EOL timelines .
>>
>> Regards
>> Pralabh Kumar
>>
>>
>>
>>
>>

Re: Spark 3.0.0 EOL

2023-07-24 Thread Hyukjin Kwon 
It's already EOL

On Mon, Jul 24, 2023 at 4:17 PM Pralabh Kumar 
wrote:

> Hi Dev Team
>
> If possible , can you please provide the Spark 3.0.0 EOL timelines .
>
> Regards
> Pralabh Kumar
>
>
>
>
>

Spark 3.0.0 EOL

2023-07-24 Thread Pralabh Kumar 
Hi Dev Team

If possible , can you please provide the Spark 3.0.0 EOL timelines .

Regards
Pralabh Kumar