Re: Moving to spark 2.4

[Tim Robertson]

To clarify Ismaël's comment

Cloudera repo indicates Cloudera 6.1 will have spark 2.4 but CDH is
currently still on 6.0.

https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.11/2.4.0-cdh6.1.0/

With the HWX / Cloudera merger the release cycle is not announced but 6.1
will likely not be there until early 2019. Also note that many folk will
still be on CDH 5 as CDH 6 is relatively new.







On Fri, Dec 7, 2018 at 5:06 PM Ismaël Mejía  wrote:

> It seems that Cloudera has it now, not sure if worth to wait for the
> Hortonworks maybe worth waiting for EMR.
>
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.11/
>
> A pro move to Spark 2.4.0 argument is for the future oriented (non
> hadoop friends), because the support for kubernetes has improved a lot
> in this release.
>
> On Fri, Dec 7, 2018 at 4:56 PM David Morávek 
> wrote:
> >
> > +1 for waiting for HDP and CDH adoption
> >
> > Sent from my iPhone
> >
> > On 7 Dec 2018, at 16:38, Alexey Romanenko 
> wrote:
> >
> > I agree with Ismael and I’d wait until the new Spark version will be
> supported by major BigData distributors.
> >
> > On 7 Dec 2018, at 14:57, Vishwas Bm  wrote:
> >
> > Hi Ismael,
> >
> > We have upgraded the spark to 2.4.
> > In our setup we had run few basic tests and found it to be pretty stable.
> >
> >
> > Thanks & Regards,
> > Vishwas
> >
> >
> > On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía  wrote:
> >>
> >> Hello Vishwas,
> >>
> >> The spark dependency in the spark runner is provided so you can
> >> already pass the dependencies of spark 2.4 and it should work out of
> >> the box.
> >>
> >> JB did a PR to upgrade the version of Spark in the runner, but maybe
> >> it is worth to wait a bit before merging it, at least until some of
> >> the Big Data distributions has spark 2.4.x support available, so far
> >> nobody has upgraded it (well apart of databricks).
> >>
> >> What do others think, should we move ahead or are you aware of any
> >> issue introduced by version 2.4.0? (Notice that the PR just updates
> >> the version so code compatibility should be ok).
> >>
> >> Ismaël
> >>
> >> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré 
> wrote:
> >> >
> >> > Hi Vishwas
> >> >
> >> > Yes, I already started the update.
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On 06/12/2018 07:39, Vishwas Bm wrote:
> >> > > Hi,
> >> > >
> >> > > Currently I see that the spark version dependency used in Beam is
> >> > > //"2.3.2".
> >> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark
> >> > > dependency ?
> >> > >
> >> > >
> >> > > *Thanks & Regards,*
> >> > > *Vishwas
> >> > > *
> >> > > *Mob : 9164886653*
> >> >
> >> > --
> >> > Jean-Baptiste Onofré
> >> > jbono...@apache.org
> >> > http://blog.nanthrax.net
> >> > Talend - http://www.talend.com
> >
> >
>

Re: Moving to spark 2.4

[Ismaël Mejía]

It seems that Cloudera has it now, not sure if worth to wait for the
Hortonworks maybe worth waiting for EMR.
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.11/

A pro move to Spark 2.4.0 argument is for the future oriented (non
hadoop friends), because the support for kubernetes has improved a lot
in this release.

On Fri, Dec 7, 2018 at 4:56 PM David Morávek  wrote:
>
> +1 for waiting for HDP and CDH adoption
>
> Sent from my iPhone
>
> On 7 Dec 2018, at 16:38, Alexey Romanenko  wrote:
>
> I agree with Ismael and I’d wait until the new Spark version will be 
> supported by major BigData distributors.
>
> On 7 Dec 2018, at 14:57, Vishwas Bm  wrote:
>
> Hi Ismael,
>
> We have upgraded the spark to 2.4.
> In our setup we had run few basic tests and found it to be pretty stable.
>
>
> Thanks & Regards,
> Vishwas
>
>
> On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía  wrote:
>>
>> Hello Vishwas,
>>
>> The spark dependency in the spark runner is provided so you can
>> already pass the dependencies of spark 2.4 and it should work out of
>> the box.
>>
>> JB did a PR to upgrade the version of Spark in the runner, but maybe
>> it is worth to wait a bit before merging it, at least until some of
>> the Big Data distributions has spark 2.4.x support available, so far
>> nobody has upgraded it (well apart of databricks).
>>
>> What do others think, should we move ahead or are you aware of any
>> issue introduced by version 2.4.0? (Notice that the PR just updates
>> the version so code compatibility should be ok).
>>
>> Ismaël
>>
>> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré  
>> wrote:
>> >
>> > Hi Vishwas
>> >
>> > Yes, I already started the update.
>> >
>> > Regards
>> > JB
>> >
>> > On 06/12/2018 07:39, Vishwas Bm wrote:
>> > > Hi,
>> > >
>> > > Currently I see that the spark version dependency used in Beam is
>> > > //"2.3.2".
>> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark
>> > > dependency ?
>> > >
>> > >
>> > > *Thanks & Regards,*
>> > > *Vishwas
>> > > *
>> > > *Mob : 9164886653*
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbono...@apache.org
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>
>

Re: Moving to spark 2.4

[David Morávek]

+1 for waiting for HDP and CDH adoption

Sent from my iPhone

> On 7 Dec 2018, at 16:38, Alexey Romanenko  wrote:
> 
> I agree with Ismael and I’d wait until the new Spark version will be 
> supported by major BigData distributors.
> 
>> On 7 Dec 2018, at 14:57, Vishwas Bm  wrote:
>> 
>> Hi Ismael,
>> 
>> We have upgraded the spark to 2.4.  
>> In our setup we had run few basic tests and found it to be pretty stable.
>> 
>> 
>> Thanks & Regards,
>> Vishwas 
>> 
>> 
>>> On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía  wrote:
>>> Hello Vishwas,
>>> 
>>> The spark dependency in the spark runner is provided so you can
>>> already pass the dependencies of spark 2.4 and it should work out of
>>> the box.
>>> 
>>> JB did a PR to upgrade the version of Spark in the runner, but maybe
>>> it is worth to wait a bit before merging it, at least until some of
>>> the Big Data distributions has spark 2.4.x support available, so far
>>> nobody has upgraded it (well apart of databricks).
>>> 
>>> What do others think, should we move ahead or are you aware of any
>>> issue introduced by version 2.4.0? (Notice that the PR just updates
>>> the version so code compatibility should be ok).
>>> 
>>> Ismaël
>>> 
>>> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré  
>>> wrote:
>>> >
>>> > Hi Vishwas
>>> >
>>> > Yes, I already started the update.
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 06/12/2018 07:39, Vishwas Bm wrote:
>>> > > Hi,
>>> > >
>>> > > Currently I see that the spark version dependency used in Beam is
>>> > > //"2.3.2".
>>> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark
>>> > > dependency ?
>>> > >
>>> > >
>>> > > *Thanks & Regards,*
>>> > > *Vishwas
>>> > > *
>>> > > *Mob : 9164886653*
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbono...@apache.org
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
> 

Re: Moving to spark 2.4

[Alexey Romanenko]

I agree with Ismael and I’d wait until the new Spark version will be supported 
by major BigData distributors.

> On 7 Dec 2018, at 14:57, Vishwas Bm  wrote:
> 
> Hi Ismael,
> 
> We have upgraded the spark to 2.4.  
> In our setup we had run few basic tests and found it to be pretty stable.
> 
> 
> Thanks & Regards,
> Vishwas 
> 
> 
> On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía  > wrote:
> Hello Vishwas,
> 
> The spark dependency in the spark runner is provided so you can
> already pass the dependencies of spark 2.4 and it should work out of
> the box.
> 
> JB did a PR to upgrade the version of Spark in the runner, but maybe
> it is worth to wait a bit before merging it, at least until some of
> the Big Data distributions has spark 2.4.x support available, so far
> nobody has upgraded it (well apart of databricks).
> 
> What do others think, should we move ahead or are you aware of any
> issue introduced by version 2.4.0? (Notice that the PR just updates
> the version so code compatibility should be ok).
> 
> Ismaël
> 
> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré  > wrote:
> >
> > Hi Vishwas
> >
> > Yes, I already started the update.
> >
> > Regards
> > JB
> >
> > On 06/12/2018 07:39, Vishwas Bm wrote:
> > > Hi,
> > >
> > > Currently I see that the spark version dependency used in Beam is
> > > //"2.3.2".
> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark
> > > dependency ?
> > >
> > >
> > > *Thanks & Regards,*
> > > *Vishwas
> > > *
> > > *Mob : 9164886653*
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org 
> > http://blog.nanthrax.net 
> > Talend - http://www.talend.com 

Re: Moving to spark 2.4

[Vishwas Bm]

Hi Ismael,

We have upgraded the spark to 2.4.
In our setup we had run few basic tests and found it to be pretty stable.


*Thanks & Regards,*

*Vishwas *


On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía  wrote:

> Hello Vishwas,
>
> The spark dependency in the spark runner is provided so you can
> already pass the dependencies of spark 2.4 and it should work out of
> the box.
>
> JB did a PR to upgrade the version of Spark in the runner, but maybe
> it is worth to wait a bit before merging it, at least until some of
> the Big Data distributions has spark 2.4.x support available, so far
> nobody has upgraded it (well apart of databricks).
>
> What do others think, should we move ahead or are you aware of any
> issue introduced by version 2.4.0? (Notice that the PR just updates
> the version so code compatibility should be ok).
>
> Ismaël
>
> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré 
> wrote:
> >
> > Hi Vishwas
> >
> > Yes, I already started the update.
> >
> > Regards
> > JB
> >
> > On 06/12/2018 07:39, Vishwas Bm wrote:
> > > Hi,
> > >
> > > Currently I see that the spark version dependency used in Beam is
> > > //"2.3.2".
> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark
> > > dependency ?
> > >
> > >
> > > *Thanks & Regards,*
> > > *Vishwas
> > > *
> > > *Mob : 9164886653*
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>

Re: Moving to spark 2.4

[Ismaël Mejía]

Hello Vishwas,

The spark dependency in the spark runner is provided so you can
already pass the dependencies of spark 2.4 and it should work out of
the box.

JB did a PR to upgrade the version of Spark in the runner, but maybe
it is worth to wait a bit before merging it, at least until some of
the Big Data distributions has spark 2.4.x support available, so far
nobody has upgraded it (well apart of databricks).

What do others think, should we move ahead or are you aware of any
issue introduced by version 2.4.0? (Notice that the PR just updates
the version so code compatibility should be ok).

Ismaël

On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré  wrote:
>
> Hi Vishwas
>
> Yes, I already started the update.
>
> Regards
> JB
>
> On 06/12/2018 07:39, Vishwas Bm wrote:
> > Hi,
> >
> > Currently I see that the spark version dependency used in Beam is
> > //"2.3.2".
> > As spark 2.4 is released now, is there a plan to upgrade Beam spark
> > dependency ?
> >
> >
> > *Thanks & Regards,*
> > *Vishwas
> > *
> > *Mob : 9164886653*
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com

Re: Moving to spark 2.4

[Jean-Baptiste Onofré]

Hi Vishwas

Yes, I already started the update.

Regards
JB

On 06/12/2018 07:39, Vishwas Bm wrote:
> Hi,
> 
> Currently I see that the spark version dependency used in Beam is
> //"2.3.2".
> As spark 2.4 is released now, is there a plan to upgrade Beam spark
> dependency ?
> 
> 
> *Thanks & Regards,*
> *Vishwas
> *
> *Mob : 9164886653*

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Moving to spark 2.4

[Vishwas Bm]

Hi,

Currently I see that the spark version dependency used in Beam is "2.3.2".
As spark 2.4 is released now, is there a plan to upgrade Beam spark
dependency ?


*Thanks & Regards,*

*Vishwas *
*Mob : 9164886653*