Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

[purna pradeep]

Thanks this is a great news

Can you please lemme if dynamic resource allocation is available in spark
2.4?

I’m using spark 2.3.2 on Kubernetes, do I still need to provide executor
memory options as part of spark submit command or spark will manage
required executor memory based on the spark job size ?

On Thu, Nov 8, 2018 at 2:18 PM Marcelo Vanzin 
wrote:

> +user@
>
> >> -- Forwarded message -
> >> From: Wenchen Fan 
> >> Date: Thu, Nov 8, 2018 at 10:55 PM
> >> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0
> >> To: Spark dev list 
> >>
> >>
> >> Hi all,
> >>
> >> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release
> adds Barrier Execution Mode for better integration with deep learning
> frameworks, introduces 30+ built-in and higher-order functions to deal with
> complex data type easier, improves the K8s integration, along with
> experimental Scala 2.12 support. Other major updates include the built-in
> Avro data source, Image data source, flexible streaming sinks, elimination
> of the 2GB block size limitation during transfer, Pandas UDF improvements.
> In addition, this release continues to focus on usability, stability, and
> polish while resolving around 1100 tickets.
> >>
> >> We'd like to thank our contributors and users for their contributions
> and early feedback to this release. This release would not have been
> possible without you.
> >>
> >> To download Spark 2.4.0, head over to the download page:
> http://spark.apache.org/downloads.html
> >>
> >> To view the release notes:
> https://spark.apache.org/releases/spark-release-2-4-0.html
> >>
> >> Thanks,
> >> Wenchen
> >>
> >> PS: If you see any issues with the release notes, webpage or published
> artifacts, please contact me directly off-list.
>
>
>
> --
> Marcelo
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

[Xiao Li]

Try to clear your browsing data or use a different web browser.

Enjoy it,

Xiao

On Thu, Nov 8, 2018 at 4:15 PM Reynold Xin  wrote:

> Do you have a cached copy? I see it here
>
> http://spark.apache.org/downloads.html
>
>
>
> On Thu, Nov 8, 2018 at 4:12 PM Li Gao  wrote:
>
>> this is wonderful !
>> I noticed the official spark download site does not have 2.4 download
>> links yet.
>>
>> On Thu, Nov 8, 2018, 4:11 PM Swapnil Shinde > wrote:
>>
>>> Great news.. thank you very much!
>>>
>>> On Thu, Nov 8, 2018, 5:19 PM Stavros Kontopoulos <
>>> stavros.kontopou...@lightbend.com wrote:
>>>
 Awesome!

 On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji 
 wrote:

> Indeed!
>
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun 
> wrote:
>
> Finally, thank you all. Especially, thanks to the release manager,
> Wenchen!
>
> Bests,
> Dongjoon.
>
>
> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan 
> wrote:
>
>> + user list
>>
>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan 
>> wrote:
>>
>>> resend
>>>
>>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan 
>>> wrote:
>>>


 -- Forwarded message -
 From: Wenchen Fan 
 Date: Thu, Nov 8, 2018 at 10:55 PM
 Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0
 To: Spark dev list 


 Hi all,

 Apache Spark 2.4.0 is the fifth release in the 2.x line. This
 release adds Barrier Execution Mode for better integration with deep
 learning frameworks, introduces 30+ built-in and higher-order 
 functions to
 deal with complex data type easier, improves the K8s integration, along
 with experimental Scala 2.12 support. Other major updates include the
 built-in Avro data source, Image data source, flexible streaming sinks,
 elimination of the 2GB block size limitation during transfer, Pandas 
 UDF
 improvements. In addition, this release continues to focus on 
 usability,
 stability, and polish while resolving around 1100 tickets.

 We'd like to thank our contributors and users for their
 contributions and early feedback to this release. This release would 
 not
 have been possible without you.

 To download Spark 2.4.0, head over to the download page:
 http://spark.apache.org/downloads.html

 To view the release notes:
 https://spark.apache.org/releases/spark-release-2-4-0.html

 Thanks,
 Wenchen

 PS: If you see any issues with the release notes, webpage or
 published artifacts, please contact me directly off-list.

>>>





-- 
[image: Spark+AI Summit North America 2019]

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

[Reynold Xin]

Do you have a cached copy? I see it here

http://spark.apache.org/downloads.html



On Thu, Nov 8, 2018 at 4:12 PM Li Gao  wrote:

> this is wonderful !
> I noticed the official spark download site does not have 2.4 download
> links yet.
>
> On Thu, Nov 8, 2018, 4:11 PM Swapnil Shinde  wrote:
>
>> Great news.. thank you very much!
>>
>> On Thu, Nov 8, 2018, 5:19 PM Stavros Kontopoulos <
>> stavros.kontopou...@lightbend.com wrote:
>>
>>> Awesome!
>>>
>>> On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji  wrote:
>>>
 Indeed!

 Sent from my iPhone
 Pardon the dumb thumb typos :)

 On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun 
 wrote:

 Finally, thank you all. Especially, thanks to the release manager,
 Wenchen!

 Bests,
 Dongjoon.


 On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan 
 wrote:

> + user list
>
> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan 
> wrote:
>
>> resend
>>
>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan 
>> wrote:
>>
>>>
>>>
>>> -- Forwarded message -
>>> From: Wenchen Fan 
>>> Date: Thu, Nov 8, 2018 at 10:55 PM
>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0
>>> To: Spark dev list 
>>>
>>>
>>> Hi all,
>>>
>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This
>>> release adds Barrier Execution Mode for better integration with deep
>>> learning frameworks, introduces 30+ built-in and higher-order functions 
>>> to
>>> deal with complex data type easier, improves the K8s integration, along
>>> with experimental Scala 2.12 support. Other major updates include the
>>> built-in Avro data source, Image data source, flexible streaming sinks,
>>> elimination of the 2GB block size limitation during transfer, Pandas UDF
>>> improvements. In addition, this release continues to focus on usability,
>>> stability, and polish while resolving around 1100 tickets.
>>>
>>> We'd like to thank our contributors and users for their
>>> contributions and early feedback to this release. This release would not
>>> have been possible without you.
>>>
>>> To download Spark 2.4.0, head over to the download page:
>>> http://spark.apache.org/downloads.html
>>>
>>> To view the release notes:
>>> https://spark.apache.org/releases/spark-release-2-4-0.html
>>>
>>> Thanks,
>>> Wenchen
>>>
>>> PS: If you see any issues with the release notes, webpage or
>>> published artifacts, please contact me directly off-list.
>>>
>>
>>>
>>>
>>>
>>>

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

[Li Gao]

this is wonderful !
I noticed the official spark download site does not have 2.4 download links
yet.

On Thu, Nov 8, 2018, 4:11 PM Swapnil Shinde  Great news.. thank you very much!
>
> On Thu, Nov 8, 2018, 5:19 PM Stavros Kontopoulos <
> stavros.kontopou...@lightbend.com wrote:
>
>> Awesome!
>>
>> On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji  wrote:
>>
>>> Indeed!
>>>
>>> Sent from my iPhone
>>> Pardon the dumb thumb typos :)
>>>
>>> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun 
>>> wrote:
>>>
>>> Finally, thank you all. Especially, thanks to the release manager,
>>> Wenchen!
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan  wrote:
>>>
 + user list

 On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan  wrote:

> resend
>
> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan 
> wrote:
>
>>
>>
>> -- Forwarded message -
>> From: Wenchen Fan 
>> Date: Thu, Nov 8, 2018 at 10:55 PM
>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0
>> To: Spark dev list 
>>
>>
>> Hi all,
>>
>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release
>> adds Barrier Execution Mode for better integration with deep learning
>> frameworks, introduces 30+ built-in and higher-order functions to deal 
>> with
>> complex data type easier, improves the K8s integration, along with
>> experimental Scala 2.12 support. Other major updates include the built-in
>> Avro data source, Image data source, flexible streaming sinks, 
>> elimination
>> of the 2GB block size limitation during transfer, Pandas UDF 
>> improvements.
>> In addition, this release continues to focus on usability, stability, and
>> polish while resolving around 1100 tickets.
>>
>> We'd like to thank our contributors and users for their contributions
>> and early feedback to this release. This release would not have been
>> possible without you.
>>
>> To download Spark 2.4.0, head over to the download page:
>> http://spark.apache.org/downloads.html
>>
>> To view the release notes:
>> https://spark.apache.org/releases/spark-release-2-4-0.html
>>
>> Thanks,
>> Wenchen
>>
>> PS: If you see any issues with the release notes, webpage or
>> published artifacts, please contact me directly off-list.
>>
>
>>
>>
>>
>>

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

[Swapnil Shinde]

Great news.. thank you very much!

On Thu, Nov 8, 2018, 5:19 PM Stavros Kontopoulos <
stavros.kontopou...@lightbend.com wrote:

> Awesome!
>
> On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji  wrote:
>
>> Indeed!
>>
>> Sent from my iPhone
>> Pardon the dumb thumb typos :)
>>
>> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun 
>> wrote:
>>
>> Finally, thank you all. Especially, thanks to the release manager,
>> Wenchen!
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan  wrote:
>>
>>> + user list
>>>
>>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan  wrote:
>>>
 resend

 On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan 
 wrote:

>
>
> -- Forwarded message -
> From: Wenchen Fan 
> Date: Thu, Nov 8, 2018 at 10:55 PM
> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0
> To: Spark dev list 
>
>
> Hi all,
>
> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release
> adds Barrier Execution Mode for better integration with deep learning
> frameworks, introduces 30+ built-in and higher-order functions to deal 
> with
> complex data type easier, improves the K8s integration, along with
> experimental Scala 2.12 support. Other major updates include the built-in
> Avro data source, Image data source, flexible streaming sinks, elimination
> of the 2GB block size limitation during transfer, Pandas UDF improvements.
> In addition, this release continues to focus on usability, stability, and
> polish while resolving around 1100 tickets.
>
> We'd like to thank our contributors and users for their contributions
> and early feedback to this release. This release would not have been
> possible without you.
>
> To download Spark 2.4.0, head over to the download page:
> http://spark.apache.org/downloads.html
>
> To view the release notes:
> https://spark.apache.org/releases/spark-release-2-4-0.html
>
> Thanks,
> Wenchen
>
> PS: If you see any issues with the release notes, webpage or published
> artifacts, please contact me directly off-list.
>

>
>
>
>

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

[Stavros Kontopoulos]

Awesome!

On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji  wrote:

> Indeed!
>
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun 
> wrote:
>
> Finally, thank you all. Especially, thanks to the release manager, Wenchen!
>
> Bests,
> Dongjoon.
>
>
> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan  wrote:
>
>> + user list
>>
>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan  wrote:
>>
>>> resend
>>>
>>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan  wrote:
>>>


 -- Forwarded message -
 From: Wenchen Fan 
 Date: Thu, Nov 8, 2018 at 10:55 PM
 Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0
 To: Spark dev list 


 Hi all,

 Apache Spark 2.4.0 is the fifth release in the 2.x line. This release
 adds Barrier Execution Mode for better integration with deep learning
 frameworks, introduces 30+ built-in and higher-order functions to deal with
 complex data type easier, improves the K8s integration, along with
 experimental Scala 2.12 support. Other major updates include the built-in
 Avro data source, Image data source, flexible streaming sinks, elimination
 of the 2GB block size limitation during transfer, Pandas UDF improvements.
 In addition, this release continues to focus on usability, stability, and
 polish while resolving around 1100 tickets.

 We'd like to thank our contributors and users for their contributions
 and early feedback to this release. This release would not have been
 possible without you.

 To download Spark 2.4.0, head over to the download page:
 http://spark.apache.org/downloads.html

 To view the release notes: https://spark.apache.org/
 releases/spark-release-2-4-0.html

 Thanks,
 Wenchen

 PS: If you see any issues with the release notes, webpage or published
 artifacts, please contact me directly off-list.

>>>

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

[Jules Damji]

Indeed! 

Sent from my iPhone
Pardon the dumb thumb typos :)

> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun  wrote:
> 
> Finally, thank you all. Especially, thanks to the release manager, Wenchen!
> 
> Bests,
> Dongjoon.
> 
> 
>> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan  wrote:
>> + user list
>> 
>>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan  wrote:
>>> resend
>>> 
 On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan  wrote:
 
 
 -- Forwarded message -
 From: Wenchen Fan 
 Date: Thu, Nov 8, 2018 at 10:55 PM
 Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0
 To: Spark dev list 
 
 
 Hi all,
 
 Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds 
 Barrier Execution Mode for better integration with deep learning 
 frameworks, introduces 30+ built-in and higher-order functions to deal 
 with complex data type easier, improves the K8s integration, along with 
 experimental Scala 2.12 support. Other major updates include the built-in 
 Avro data source, Image data source, flexible streaming sinks, elimination 
 of the 2GB block size limitation during transfer, Pandas UDF improvements. 
 In addition, this release continues to focus on usability, stability, and 
 polish while resolving around 1100 tickets.
 
 We'd like to thank our contributors and users for their contributions and 
 early feedback to this release. This release would not have been possible 
 without you.
 
 To download Spark 2.4.0, head over to the download page: 
 http://spark.apache.org/downloads.html
 
 To view the release notes: 
 https://spark.apache.org/releases/spark-release-2-4-0.html
 
 Thanks,
 Wenchen
 
 PS: If you see any issues with the release notes, webpage or published 
 artifacts, please contact me directly off-list.

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

[Dongjoon Hyun]

Finally, thank you all. Especially, thanks to the release manager, Wenchen!

Bests,
Dongjoon.


On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan  wrote:

> + user list
>
> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan  wrote:
>
>> resend
>>
>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan  wrote:
>>
>>>
>>>
>>> -- Forwarded message -
>>> From: Wenchen Fan 
>>> Date: Thu, Nov 8, 2018 at 10:55 PM
>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0
>>> To: Spark dev list 
>>>
>>>
>>> Hi all,
>>>
>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release
>>> adds Barrier Execution Mode for better integration with deep learning
>>> frameworks, introduces 30+ built-in and higher-order functions to deal with
>>> complex data type easier, improves the K8s integration, along with
>>> experimental Scala 2.12 support. Other major updates include the built-in
>>> Avro data source, Image data source, flexible streaming sinks, elimination
>>> of the 2GB block size limitation during transfer, Pandas UDF improvements.
>>> In addition, this release continues to focus on usability, stability, and
>>> polish while resolving around 1100 tickets.
>>>
>>> We'd like to thank our contributors and users for their contributions
>>> and early feedback to this release. This release would not have been
>>> possible without you.
>>>
>>> To download Spark 2.4.0, head over to the download page:
>>> http://spark.apache.org/downloads.html
>>>
>>> To view the release notes:
>>> https://spark.apache.org/releases/spark-release-2-4-0.html
>>>
>>> Thanks,
>>> Wenchen
>>>
>>> PS: If you see any issues with the release notes, webpage or published
>>> artifacts, please contact me directly off-list.
>>>
>>

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

[Wenchen Fan]

+ user list

On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan  wrote:

> resend
>
> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan  wrote:
>
>>
>>
>> -- Forwarded message -
>> From: Wenchen Fan 
>> Date: Thu, Nov 8, 2018 at 10:55 PM
>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0
>> To: Spark dev list 
>>
>>
>> Hi all,
>>
>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release
>> adds Barrier Execution Mode for better integration with deep learning
>> frameworks, introduces 30+ built-in and higher-order functions to deal with
>> complex data type easier, improves the K8s integration, along with
>> experimental Scala 2.12 support. Other major updates include the built-in
>> Avro data source, Image data source, flexible streaming sinks, elimination
>> of the 2GB block size limitation during transfer, Pandas UDF improvements.
>> In addition, this release continues to focus on usability, stability, and
>> polish while resolving around 1100 tickets.
>>
>> We'd like to thank our contributors and users for their contributions and
>> early feedback to this release. This release would not have been possible
>> without you.
>>
>> To download Spark 2.4.0, head over to the download page:
>> http://spark.apache.org/downloads.html
>>
>> To view the release notes:
>> https://spark.apache.org/releases/spark-release-2-4-0.html
>>
>> Thanks,
>> Wenchen
>>
>> PS: If you see any issues with the release notes, webpage or published
>> artifacts, please contact me directly off-list.
>>
>

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

[Marcelo Vanzin]

+user@

>> -- Forwarded message -
>> From: Wenchen Fan 
>> Date: Thu, Nov 8, 2018 at 10:55 PM
>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0
>> To: Spark dev list 
>>
>>
>> Hi all,
>>
>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds 
>> Barrier Execution Mode for better integration with deep learning frameworks, 
>> introduces 30+ built-in and higher-order functions to deal with complex data 
>> type easier, improves the K8s integration, along with experimental Scala 
>> 2.12 support. Other major updates include the built-in Avro data source, 
>> Image data source, flexible streaming sinks, elimination of the 2GB block 
>> size limitation during transfer, Pandas UDF improvements. In addition, this 
>> release continues to focus on usability, stability, and polish while 
>> resolving around 1100 tickets.
>>
>> We'd like to thank our contributors and users for their contributions and 
>> early feedback to this release. This release would not have been possible 
>> without you.
>>
>> To download Spark 2.4.0, head over to the download page: 
>> http://spark.apache.org/downloads.html
>>
>> To view the release notes: 
>> https://spark.apache.org/releases/spark-release-2-4-0.html
>>
>> Thanks,
>> Wenchen
>>
>> PS: If you see any issues with the release notes, webpage or published 
>> artifacts, please contact me directly off-list.



-- 
Marcelo

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org