Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Thanks this is a great news Can you please lemme if dynamic resource allocation is available in spark 2.4? I’m using spark 2.3.2 on Kubernetes, do I still need to provide executor memory options as part of spark submit command or spark will manage required executor memory based on the spark job size ? On Thu, Nov 8, 2018 at 2:18 PM Marcelo Vanzin wrote: > +user@ > > >> -- Forwarded message - > >> From: Wenchen Fan > >> Date: Thu, Nov 8, 2018 at 10:55 PM > >> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 > >> To: Spark dev list > >> > >> > >> Hi all, > >> > >> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release > adds Barrier Execution Mode for better integration with deep learning > frameworks, introduces 30+ built-in and higher-order functions to deal with > complex data type easier, improves the K8s integration, along with > experimental Scala 2.12 support. Other major updates include the built-in > Avro data source, Image data source, flexible streaming sinks, elimination > of the 2GB block size limitation during transfer, Pandas UDF improvements. > In addition, this release continues to focus on usability, stability, and > polish while resolving around 1100 tickets. > >> > >> We'd like to thank our contributors and users for their contributions > and early feedback to this release. This release would not have been > possible without you. > >> > >> To download Spark 2.4.0, head over to the download page: > http://spark.apache.org/downloads.html > >> > >> To view the release notes: > https://spark.apache.org/releases/spark-release-2-4-0.html > >> > >> Thanks, > >> Wenchen > >> > >> PS: If you see any issues with the release notes, webpage or published > artifacts, please contact me directly off-list. > > > > -- > Marcelo > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Try to clear your browsing data or use a different web browser. Enjoy it, Xiao On Thu, Nov 8, 2018 at 4:15 PM Reynold Xin wrote: > Do you have a cached copy? I see it here > > http://spark.apache.org/downloads.html > > > > On Thu, Nov 8, 2018 at 4:12 PM Li Gao wrote: > >> this is wonderful ! >> I noticed the official spark download site does not have 2.4 download >> links yet. >> >> On Thu, Nov 8, 2018, 4:11 PM Swapnil Shinde > wrote: >> >>> Great news.. thank you very much! >>> >>> On Thu, Nov 8, 2018, 5:19 PM Stavros Kontopoulos < >>> stavros.kontopou...@lightbend.com wrote: >>> >>>> Awesome! >>>> >>>> On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji >>>> wrote: >>>> >>>>> Indeed! >>>>> >>>>> Sent from my iPhone >>>>> Pardon the dumb thumb typos :) >>>>> >>>>> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun >>>>> wrote: >>>>> >>>>> Finally, thank you all. Especially, thanks to the release manager, >>>>> Wenchen! >>>>> >>>>> Bests, >>>>> Dongjoon. >>>>> >>>>> >>>>> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan >>>>> wrote: >>>>> >>>>>> + user list >>>>>> >>>>>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan >>>>>> wrote: >>>>>> >>>>>>> resend >>>>>>> >>>>>>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- Forwarded message - >>>>>>>> From: Wenchen Fan >>>>>>>> Date: Thu, Nov 8, 2018 at 10:55 PM >>>>>>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>>>>>>> To: Spark dev list >>>>>>>> >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This >>>>>>>> release adds Barrier Execution Mode for better integration with deep >>>>>>>> learning frameworks, introduces 30+ built-in and higher-order >>>>>>>> functions to >>>>>>>> deal with complex data type easier, improves the K8s integration, along >>>>>>>> with experimental Scala 2.12 support. Other major updates include the >>>>>>>> built-in Avro data source, Image data source, flexible streaming sinks, >>>>>>>> elimination of the 2GB block size limitation during transfer, Pandas >>>>>>>> UDF >>>>>>>> improvements. In addition, this release continues to focus on >>>>>>>> usability, >>>>>>>> stability, and polish while resolving around 1100 tickets. >>>>>>>> >>>>>>>> We'd like to thank our contributors and users for their >>>>>>>> contributions and early feedback to this release. This release would >>>>>>>> not >>>>>>>> have been possible without you. >>>>>>>> >>>>>>>> To download Spark 2.4.0, head over to the download page: >>>>>>>> http://spark.apache.org/downloads.html >>>>>>>> >>>>>>>> To view the release notes: >>>>>>>> https://spark.apache.org/releases/spark-release-2-4-0.html >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Wenchen >>>>>>>> >>>>>>>> PS: If you see any issues with the release notes, webpage or >>>>>>>> published artifacts, please contact me directly off-list. >>>>>>>> >>>>>>> >>>> >>>> >>>> >>>> -- [image: Spark+AI Summit North America 2019] <http://t.sidekickopen24.com/s1t/c/5/f18dQhb0S7lM8dDMPbW2n0x6l2B9nMJN7t5X-FfhMynN2z8MDjQsyTKW56dzQQ1-_gV6102?t=https%3A%2F%2Fdatabricks.com%2Fsparkaisummit%2Fnorth-america=undefined=406b8c9a-b648-4923-9ed1-9a51ffe213fa>
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Do you have a cached copy? I see it here http://spark.apache.org/downloads.html On Thu, Nov 8, 2018 at 4:12 PM Li Gao wrote: > this is wonderful ! > I noticed the official spark download site does not have 2.4 download > links yet. > > On Thu, Nov 8, 2018, 4:11 PM Swapnil Shinde wrote: > >> Great news.. thank you very much! >> >> On Thu, Nov 8, 2018, 5:19 PM Stavros Kontopoulos < >> stavros.kontopou...@lightbend.com wrote: >> >>> Awesome! >>> >>> On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji wrote: >>> >>>> Indeed! >>>> >>>> Sent from my iPhone >>>> Pardon the dumb thumb typos :) >>>> >>>> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun >>>> wrote: >>>> >>>> Finally, thank you all. Especially, thanks to the release manager, >>>> Wenchen! >>>> >>>> Bests, >>>> Dongjoon. >>>> >>>> >>>> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan >>>> wrote: >>>> >>>>> + user list >>>>> >>>>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan >>>>> wrote: >>>>> >>>>>> resend >>>>>> >>>>>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> -- Forwarded message - >>>>>>> From: Wenchen Fan >>>>>>> Date: Thu, Nov 8, 2018 at 10:55 PM >>>>>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>>>>>> To: Spark dev list >>>>>>> >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This >>>>>>> release adds Barrier Execution Mode for better integration with deep >>>>>>> learning frameworks, introduces 30+ built-in and higher-order functions >>>>>>> to >>>>>>> deal with complex data type easier, improves the K8s integration, along >>>>>>> with experimental Scala 2.12 support. Other major updates include the >>>>>>> built-in Avro data source, Image data source, flexible streaming sinks, >>>>>>> elimination of the 2GB block size limitation during transfer, Pandas UDF >>>>>>> improvements. In addition, this release continues to focus on usability, >>>>>>> stability, and polish while resolving around 1100 tickets. >>>>>>> >>>>>>> We'd like to thank our contributors and users for their >>>>>>> contributions and early feedback to this release. This release would not >>>>>>> have been possible without you. >>>>>>> >>>>>>> To download Spark 2.4.0, head over to the download page: >>>>>>> http://spark.apache.org/downloads.html >>>>>>> >>>>>>> To view the release notes: >>>>>>> https://spark.apache.org/releases/spark-release-2-4-0.html >>>>>>> >>>>>>> Thanks, >>>>>>> Wenchen >>>>>>> >>>>>>> PS: If you see any issues with the release notes, webpage or >>>>>>> published artifacts, please contact me directly off-list. >>>>>>> >>>>>> >>> >>> >>> >>>
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
this is wonderful ! I noticed the official spark download site does not have 2.4 download links yet. On Thu, Nov 8, 2018, 4:11 PM Swapnil Shinde Great news.. thank you very much! > > On Thu, Nov 8, 2018, 5:19 PM Stavros Kontopoulos < > stavros.kontopou...@lightbend.com wrote: > >> Awesome! >> >> On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji wrote: >> >>> Indeed! >>> >>> Sent from my iPhone >>> Pardon the dumb thumb typos :) >>> >>> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun >>> wrote: >>> >>> Finally, thank you all. Especially, thanks to the release manager, >>> Wenchen! >>> >>> Bests, >>> Dongjoon. >>> >>> >>> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan wrote: >>> >>>> + user list >>>> >>>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote: >>>> >>>>> resend >>>>> >>>>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> -- Forwarded message - >>>>>> From: Wenchen Fan >>>>>> Date: Thu, Nov 8, 2018 at 10:55 PM >>>>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>>>>> To: Spark dev list >>>>>> >>>>>> >>>>>> Hi all, >>>>>> >>>>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release >>>>>> adds Barrier Execution Mode for better integration with deep learning >>>>>> frameworks, introduces 30+ built-in and higher-order functions to deal >>>>>> with >>>>>> complex data type easier, improves the K8s integration, along with >>>>>> experimental Scala 2.12 support. Other major updates include the built-in >>>>>> Avro data source, Image data source, flexible streaming sinks, >>>>>> elimination >>>>>> of the 2GB block size limitation during transfer, Pandas UDF >>>>>> improvements. >>>>>> In addition, this release continues to focus on usability, stability, and >>>>>> polish while resolving around 1100 tickets. >>>>>> >>>>>> We'd like to thank our contributors and users for their contributions >>>>>> and early feedback to this release. This release would not have been >>>>>> possible without you. >>>>>> >>>>>> To download Spark 2.4.0, head over to the download page: >>>>>> http://spark.apache.org/downloads.html >>>>>> >>>>>> To view the release notes: >>>>>> https://spark.apache.org/releases/spark-release-2-4-0.html >>>>>> >>>>>> Thanks, >>>>>> Wenchen >>>>>> >>>>>> PS: If you see any issues with the release notes, webpage or >>>>>> published artifacts, please contact me directly off-list. >>>>>> >>>>> >> >> >> >>
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Great news.. thank you very much! On Thu, Nov 8, 2018, 5:19 PM Stavros Kontopoulos < stavros.kontopou...@lightbend.com wrote: > Awesome! > > On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji wrote: > >> Indeed! >> >> Sent from my iPhone >> Pardon the dumb thumb typos :) >> >> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun >> wrote: >> >> Finally, thank you all. Especially, thanks to the release manager, >> Wenchen! >> >> Bests, >> Dongjoon. >> >> >> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan wrote: >> >>> + user list >>> >>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote: >>> >>>> resend >>>> >>>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan >>>> wrote: >>>> >>>>> >>>>> >>>>> -- Forwarded message - >>>>> From: Wenchen Fan >>>>> Date: Thu, Nov 8, 2018 at 10:55 PM >>>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>>>> To: Spark dev list >>>>> >>>>> >>>>> Hi all, >>>>> >>>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release >>>>> adds Barrier Execution Mode for better integration with deep learning >>>>> frameworks, introduces 30+ built-in and higher-order functions to deal >>>>> with >>>>> complex data type easier, improves the K8s integration, along with >>>>> experimental Scala 2.12 support. Other major updates include the built-in >>>>> Avro data source, Image data source, flexible streaming sinks, elimination >>>>> of the 2GB block size limitation during transfer, Pandas UDF improvements. >>>>> In addition, this release continues to focus on usability, stability, and >>>>> polish while resolving around 1100 tickets. >>>>> >>>>> We'd like to thank our contributors and users for their contributions >>>>> and early feedback to this release. This release would not have been >>>>> possible without you. >>>>> >>>>> To download Spark 2.4.0, head over to the download page: >>>>> http://spark.apache.org/downloads.html >>>>> >>>>> To view the release notes: >>>>> https://spark.apache.org/releases/spark-release-2-4-0.html >>>>> >>>>> Thanks, >>>>> Wenchen >>>>> >>>>> PS: If you see any issues with the release notes, webpage or published >>>>> artifacts, please contact me directly off-list. >>>>> >>>> > > > >
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Awesome! On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji wrote: > Indeed! > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun > wrote: > > Finally, thank you all. Especially, thanks to the release manager, Wenchen! > > Bests, > Dongjoon. > > > On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan wrote: > >> + user list >> >> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote: >> >>> resend >>> >>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan wrote: >>> >>>> >>>> >>>> ------ Forwarded message - >>>> From: Wenchen Fan >>>> Date: Thu, Nov 8, 2018 at 10:55 PM >>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>>> To: Spark dev list >>>> >>>> >>>> Hi all, >>>> >>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release >>>> adds Barrier Execution Mode for better integration with deep learning >>>> frameworks, introduces 30+ built-in and higher-order functions to deal with >>>> complex data type easier, improves the K8s integration, along with >>>> experimental Scala 2.12 support. Other major updates include the built-in >>>> Avro data source, Image data source, flexible streaming sinks, elimination >>>> of the 2GB block size limitation during transfer, Pandas UDF improvements. >>>> In addition, this release continues to focus on usability, stability, and >>>> polish while resolving around 1100 tickets. >>>> >>>> We'd like to thank our contributors and users for their contributions >>>> and early feedback to this release. This release would not have been >>>> possible without you. >>>> >>>> To download Spark 2.4.0, head over to the download page: >>>> http://spark.apache.org/downloads.html >>>> >>>> To view the release notes: https://spark.apache.org/ >>>> releases/spark-release-2-4-0.html >>>> >>>> Thanks, >>>> Wenchen >>>> >>>> PS: If you see any issues with the release notes, webpage or published >>>> artifacts, please contact me directly off-list. >>>> >>>
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Indeed! Sent from my iPhone Pardon the dumb thumb typos :) > On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun wrote: > > Finally, thank you all. Especially, thanks to the release manager, Wenchen! > > Bests, > Dongjoon. > > >> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan wrote: >> + user list >> >>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote: >>> resend >>> >>>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan wrote: >>>> >>>> >>>> -- Forwarded message ----- >>>> From: Wenchen Fan >>>> Date: Thu, Nov 8, 2018 at 10:55 PM >>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>>> To: Spark dev list >>>> >>>> >>>> Hi all, >>>> >>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds >>>> Barrier Execution Mode for better integration with deep learning >>>> frameworks, introduces 30+ built-in and higher-order functions to deal >>>> with complex data type easier, improves the K8s integration, along with >>>> experimental Scala 2.12 support. Other major updates include the built-in >>>> Avro data source, Image data source, flexible streaming sinks, elimination >>>> of the 2GB block size limitation during transfer, Pandas UDF improvements. >>>> In addition, this release continues to focus on usability, stability, and >>>> polish while resolving around 1100 tickets. >>>> >>>> We'd like to thank our contributors and users for their contributions and >>>> early feedback to this release. This release would not have been possible >>>> without you. >>>> >>>> To download Spark 2.4.0, head over to the download page: >>>> http://spark.apache.org/downloads.html >>>> >>>> To view the release notes: >>>> https://spark.apache.org/releases/spark-release-2-4-0.html >>>> >>>> Thanks, >>>> Wenchen >>>> >>>> PS: If you see any issues with the release notes, webpage or published >>>> artifacts, please contact me directly off-list.
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Finally, thank you all. Especially, thanks to the release manager, Wenchen! Bests, Dongjoon. On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan wrote: > + user list > > On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote: > >> resend >> >> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan wrote: >> >>> >>> >>> -- Forwarded message - >>> From: Wenchen Fan >>> Date: Thu, Nov 8, 2018 at 10:55 PM >>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>> To: Spark dev list >>> >>> >>> Hi all, >>> >>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release >>> adds Barrier Execution Mode for better integration with deep learning >>> frameworks, introduces 30+ built-in and higher-order functions to deal with >>> complex data type easier, improves the K8s integration, along with >>> experimental Scala 2.12 support. Other major updates include the built-in >>> Avro data source, Image data source, flexible streaming sinks, elimination >>> of the 2GB block size limitation during transfer, Pandas UDF improvements. >>> In addition, this release continues to focus on usability, stability, and >>> polish while resolving around 1100 tickets. >>> >>> We'd like to thank our contributors and users for their contributions >>> and early feedback to this release. This release would not have been >>> possible without you. >>> >>> To download Spark 2.4.0, head over to the download page: >>> http://spark.apache.org/downloads.html >>> >>> To view the release notes: >>> https://spark.apache.org/releases/spark-release-2-4-0.html >>> >>> Thanks, >>> Wenchen >>> >>> PS: If you see any issues with the release notes, webpage or published >>> artifacts, please contact me directly off-list. >>> >>
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
+ user list On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote: > resend > > On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan wrote: > >> >> >> -- Forwarded message - >> From: Wenchen Fan >> Date: Thu, Nov 8, 2018 at 10:55 PM >> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >> To: Spark dev list >> >> >> Hi all, >> >> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release >> adds Barrier Execution Mode for better integration with deep learning >> frameworks, introduces 30+ built-in and higher-order functions to deal with >> complex data type easier, improves the K8s integration, along with >> experimental Scala 2.12 support. Other major updates include the built-in >> Avro data source, Image data source, flexible streaming sinks, elimination >> of the 2GB block size limitation during transfer, Pandas UDF improvements. >> In addition, this release continues to focus on usability, stability, and >> polish while resolving around 1100 tickets. >> >> We'd like to thank our contributors and users for their contributions and >> early feedback to this release. This release would not have been possible >> without you. >> >> To download Spark 2.4.0, head over to the download page: >> http://spark.apache.org/downloads.html >> >> To view the release notes: >> https://spark.apache.org/releases/spark-release-2-4-0.html >> >> Thanks, >> Wenchen >> >> PS: If you see any issues with the release notes, webpage or published >> artifacts, please contact me directly off-list. >> >
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
+user@ >> -- Forwarded message - >> From: Wenchen Fan >> Date: Thu, Nov 8, 2018 at 10:55 PM >> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >> To: Spark dev list >> >> >> Hi all, >> >> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds >> Barrier Execution Mode for better integration with deep learning frameworks, >> introduces 30+ built-in and higher-order functions to deal with complex data >> type easier, improves the K8s integration, along with experimental Scala >> 2.12 support. Other major updates include the built-in Avro data source, >> Image data source, flexible streaming sinks, elimination of the 2GB block >> size limitation during transfer, Pandas UDF improvements. In addition, this >> release continues to focus on usability, stability, and polish while >> resolving around 1100 tickets. >> >> We'd like to thank our contributors and users for their contributions and >> early feedback to this release. This release would not have been possible >> without you. >> >> To download Spark 2.4.0, head over to the download page: >> http://spark.apache.org/downloads.html >> >> To view the release notes: >> https://spark.apache.org/releases/spark-release-2-4-0.html >> >> Thanks, >> Wenchen >> >> PS: If you see any issues with the release notes, webpage or published >> artifacts, please contact me directly off-list. -- Marcelo - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
