Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Thanks this is a great news Can you please lemme if dynamic resource allocation is available in spark 2.4? I’m using spark 2.3.2 on Kubernetes, do I still need to provide executor memory options as part of spark submit command or spark will manage required executor memory based on the spark job size ? On Thu, Nov 8, 2018 at 2:18 PM Marcelo Vanzin wrote: > +user@ > > >> -- Forwarded message - > >> From: Wenchen Fan > >> Date: Thu, Nov 8, 2018 at 10:55 PM > >> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 > >> To: Spark dev list > >> > >> > >> Hi all, > >> > >> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release > adds Barrier Execution Mode for better integration with deep learning > frameworks, introduces 30+ built-in and higher-order functions to deal with > complex data type easier, improves the K8s integration, along with > experimental Scala 2.12 support. Other major updates include the built-in > Avro data source, Image data source, flexible streaming sinks, elimination > of the 2GB block size limitation during transfer, Pandas UDF improvements. > In addition, this release continues to focus on usability, stability, and > polish while resolving around 1100 tickets. > >> > >> We'd like to thank our contributors and users for their contributions > and early feedback to this release. This release would not have been > possible without you. > >> > >> To download Spark 2.4.0, head over to the download page: > http://spark.apache.org/downloads.html > >> > >> To view the release notes: > https://spark.apache.org/releases/spark-release-2-4-0.html > >> > >> Thanks, > >> Wenchen > >> > >> PS: If you see any issues with the release notes, webpage or published > artifacts, please contact me directly off-list. > > > > -- > Marcelo > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Try to clear your browsing data or use a different web browser. Enjoy it, Xiao On Thu, Nov 8, 2018 at 4:15 PM Reynold Xin wrote: > Do you have a cached copy? I see it here > > http://spark.apache.org/downloads.html > > > > On Thu, Nov 8, 2018 at 4:12 PM Li Gao wrote: > >> this is wonderful ! >> I noticed the official spark download site does not have 2.4 download >> links yet. >> >> On Thu, Nov 8, 2018, 4:11 PM Swapnil Shinde > wrote: >> >>> Great news.. thank you very much! >>> >>> On Thu, Nov 8, 2018, 5:19 PM Stavros Kontopoulos < >>> stavros.kontopou...@lightbend.com wrote: >>> >>>> Awesome! >>>> >>>> On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji >>>> wrote: >>>> >>>>> Indeed! >>>>> >>>>> Sent from my iPhone >>>>> Pardon the dumb thumb typos :) >>>>> >>>>> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun >>>>> wrote: >>>>> >>>>> Finally, thank you all. Especially, thanks to the release manager, >>>>> Wenchen! >>>>> >>>>> Bests, >>>>> Dongjoon. >>>>> >>>>> >>>>> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan >>>>> wrote: >>>>> >>>>>> + user list >>>>>> >>>>>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan >>>>>> wrote: >>>>>> >>>>>>> resend >>>>>>> >>>>>>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- Forwarded message - >>>>>>>> From: Wenchen Fan >>>>>>>> Date: Thu, Nov 8, 2018 at 10:55 PM >>>>>>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>>>>>>> To: Spark dev list >>>>>>>> >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This >>>>>>>> release adds Barrier Execution Mode for better integration with deep >>>>>>>> learning frameworks, introduces 30+ built-in and higher-order >>>>>>>> functions to >>>>>>>> deal with complex data type easier, improves the K8s integration, along >>>>>>>> with experimental Scala 2.12 support. Other major updates include the >>>>>>>> built-in Avro data source, Image data source, flexible streaming sinks, >>>>>>>> elimination of the 2GB block size limitation during transfer, Pandas >>>>>>>> UDF >>>>>>>> improvements. In addition, this release continues to focus on >>>>>>>> usability, >>>>>>>> stability, and polish while resolving around 1100 tickets. >>>>>>>> >>>>>>>> We'd like to thank our contributors and users for their >>>>>>>> contributions and early feedback to this release. This release would >>>>>>>> not >>>>>>>> have been possible without you. >>>>>>>> >>>>>>>> To download Spark 2.4.0, head over to the download page: >>>>>>>> http://spark.apache.org/downloads.html >>>>>>>> >>>>>>>> To view the release notes: >>>>>>>> https://spark.apache.org/releases/spark-release-2-4-0.html >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Wenchen >>>>>>>> >>>>>>>> PS: If you see any issues with the release notes, webpage or >>>>>>>> published artifacts, please contact me directly off-list. >>>>>>>> >>>>>>> >>>> >>>> >>>> >>>> -- [image: Spark+AI Summit North America 2019] <http://t.sidekickopen24.com/s1t/c/5/f18dQhb0S7lM8dDMPbW2n0x6l2B9nMJN7t5X-FfhMynN2z8MDjQsyTKW56dzQQ1-_gV6102?t=https%3A%2F%2Fdatabricks.com%2Fsparkaisummit%2Fnorth-america=undefined=406b8c9a-b648-4923-9ed1-9a51ffe213fa>
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Do you have a cached copy? I see it here http://spark.apache.org/downloads.html On Thu, Nov 8, 2018 at 4:12 PM Li Gao wrote: > this is wonderful ! > I noticed the official spark download site does not have 2.4 download > links yet. > > On Thu, Nov 8, 2018, 4:11 PM Swapnil Shinde wrote: > >> Great news.. thank you very much! >> >> On Thu, Nov 8, 2018, 5:19 PM Stavros Kontopoulos < >> stavros.kontopou...@lightbend.com wrote: >> >>> Awesome! >>> >>> On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji wrote: >>> >>>> Indeed! >>>> >>>> Sent from my iPhone >>>> Pardon the dumb thumb typos :) >>>> >>>> On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun >>>> wrote: >>>> >>>> Finally, thank you all. Especially, thanks to the release manager, >>>> Wenchen! >>>> >>>> Bests, >>>> Dongjoon. >>>> >>>> >>>> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan >>>> wrote: >>>> >>>>> + user list >>>>> >>>>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan >>>>> wrote: >>>>> >>>>>> resend >>>>>> >>>>>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> -- Forwarded message - >>>>>>> From: Wenchen Fan >>>>>>> Date: Thu, Nov 8, 2018 at 10:55 PM >>>>>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>>>>>> To: Spark dev list >>>>>>> >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This >>>>>>> release adds Barrier Execution Mode for better integration with deep >>>>>>> learning frameworks, introduces 30+ built-in and higher-order functions >>>>>>> to >>>>>>> deal with complex data type easier, improves the K8s integration, along >>>>>>> with experimental Scala 2.12 support. Other major updates include the >>>>>>> built-in Avro data source, Image data source, flexible streaming sinks, >>>>>>> elimination of the 2GB block size limitation during transfer, Pandas UDF >>>>>>> improvements. In addition, this release continues to focus on usability, >>>>>>> stability, and polish while resolving around 1100 tickets. >>>>>>> >>>>>>> We'd like to thank our contributors and users for their >>>>>>> contributions and early feedback to this release. This release would not >>>>>>> have been possible without you. >>>>>>> >>>>>>> To download Spark 2.4.0, head over to the download page: >>>>>>> http://spark.apache.org/downloads.html >>>>>>> >>>>>>> To view the release notes: >>>>>>> https://spark.apache.org/releases/spark-release-2-4-0.html >>>>>>> >>>>>>> Thanks, >>>>>>> Wenchen >>>>>>> >>>>>>> PS: If you see any issues with the release notes, webpage or >>>>>>> published artifacts, please contact me directly off-list. >>>>>>> >>>>>> >>> >>> >>> >>>
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Awesome! On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji wrote: > Indeed! > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun > wrote: > > Finally, thank you all. Especially, thanks to the release manager, Wenchen! > > Bests, > Dongjoon. > > > On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan wrote: > >> + user list >> >> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote: >> >>> resend >>> >>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan wrote: >>> >>>> >>>> >>>> ------ Forwarded message - >>>> From: Wenchen Fan >>>> Date: Thu, Nov 8, 2018 at 10:55 PM >>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>>> To: Spark dev list >>>> >>>> >>>> Hi all, >>>> >>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release >>>> adds Barrier Execution Mode for better integration with deep learning >>>> frameworks, introduces 30+ built-in and higher-order functions to deal with >>>> complex data type easier, improves the K8s integration, along with >>>> experimental Scala 2.12 support. Other major updates include the built-in >>>> Avro data source, Image data source, flexible streaming sinks, elimination >>>> of the 2GB block size limitation during transfer, Pandas UDF improvements. >>>> In addition, this release continues to focus on usability, stability, and >>>> polish while resolving around 1100 tickets. >>>> >>>> We'd like to thank our contributors and users for their contributions >>>> and early feedback to this release. This release would not have been >>>> possible without you. >>>> >>>> To download Spark 2.4.0, head over to the download page: >>>> http://spark.apache.org/downloads.html >>>> >>>> To view the release notes: https://spark.apache.org/ >>>> releases/spark-release-2-4-0.html >>>> >>>> Thanks, >>>> Wenchen >>>> >>>> PS: If you see any issues with the release notes, webpage or published >>>> artifacts, please contact me directly off-list. >>>> >>>
[ANNOUNCE] Announcing Apache Spark 2.4.0
Hi all, Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds Barrier Execution Mode for better integration with deep learning frameworks, introduces 30+ built-in and higher-order functions to deal with complex data type easier, improves the K8s integration, along with experimental Scala 2.12 support. Other major updates include the built-in Avro data source, Image data source, flexible streaming sinks, elimination of the 2GB block size limitation during transfer, Pandas UDF improvements. In addition, this release continues to focus on usability, stability, and polish while resolving around 1100 tickets. We'd like to thank our contributors and users for their contributions and early feedback to this release. This release would not have been possible without you. To download Spark 2.4.0, head over to the download page: http://spark.apache.org/downloads.html To view the release notes: https://spark.apache.org/releases/spark-release-2-4-0.html Thanks, Wenchen PS: If you see any issues with the release notes, webpage or published artifacts, please contact me directly off-list.
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Indeed! Sent from my iPhone Pardon the dumb thumb typos :) > On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun wrote: > > Finally, thank you all. Especially, thanks to the release manager, Wenchen! > > Bests, > Dongjoon. > > >> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan wrote: >> + user list >> >>> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote: >>> resend >>> >>>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan wrote: >>>> >>>> >>>> -- Forwarded message ----- >>>> From: Wenchen Fan >>>> Date: Thu, Nov 8, 2018 at 10:55 PM >>>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>>> To: Spark dev list >>>> >>>> >>>> Hi all, >>>> >>>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds >>>> Barrier Execution Mode for better integration with deep learning >>>> frameworks, introduces 30+ built-in and higher-order functions to deal >>>> with complex data type easier, improves the K8s integration, along with >>>> experimental Scala 2.12 support. Other major updates include the built-in >>>> Avro data source, Image data source, flexible streaming sinks, elimination >>>> of the 2GB block size limitation during transfer, Pandas UDF improvements. >>>> In addition, this release continues to focus on usability, stability, and >>>> polish while resolving around 1100 tickets. >>>> >>>> We'd like to thank our contributors and users for their contributions and >>>> early feedback to this release. This release would not have been possible >>>> without you. >>>> >>>> To download Spark 2.4.0, head over to the download page: >>>> http://spark.apache.org/downloads.html >>>> >>>> To view the release notes: >>>> https://spark.apache.org/releases/spark-release-2-4-0.html >>>> >>>> Thanks, >>>> Wenchen >>>> >>>> PS: If you see any issues with the release notes, webpage or published >>>> artifacts, please contact me directly off-list.
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
Finally, thank you all. Especially, thanks to the release manager, Wenchen! Bests, Dongjoon. On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan wrote: > + user list > > On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote: > >> resend >> >> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan wrote: >> >>> >>> >>> -- Forwarded message - >>> From: Wenchen Fan >>> Date: Thu, Nov 8, 2018 at 10:55 PM >>> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >>> To: Spark dev list >>> >>> >>> Hi all, >>> >>> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release >>> adds Barrier Execution Mode for better integration with deep learning >>> frameworks, introduces 30+ built-in and higher-order functions to deal with >>> complex data type easier, improves the K8s integration, along with >>> experimental Scala 2.12 support. Other major updates include the built-in >>> Avro data source, Image data source, flexible streaming sinks, elimination >>> of the 2GB block size limitation during transfer, Pandas UDF improvements. >>> In addition, this release continues to focus on usability, stability, and >>> polish while resolving around 1100 tickets. >>> >>> We'd like to thank our contributors and users for their contributions >>> and early feedback to this release. This release would not have been >>> possible without you. >>> >>> To download Spark 2.4.0, head over to the download page: >>> http://spark.apache.org/downloads.html >>> >>> To view the release notes: >>> https://spark.apache.org/releases/spark-release-2-4-0.html >>> >>> Thanks, >>> Wenchen >>> >>> PS: If you see any issues with the release notes, webpage or published >>> artifacts, please contact me directly off-list. >>> >>
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
+ user list On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote: > resend > > On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan wrote: > >> >> >> -- Forwarded message - >> From: Wenchen Fan >> Date: Thu, Nov 8, 2018 at 10:55 PM >> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >> To: Spark dev list >> >> >> Hi all, >> >> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release >> adds Barrier Execution Mode for better integration with deep learning >> frameworks, introduces 30+ built-in and higher-order functions to deal with >> complex data type easier, improves the K8s integration, along with >> experimental Scala 2.12 support. Other major updates include the built-in >> Avro data source, Image data source, flexible streaming sinks, elimination >> of the 2GB block size limitation during transfer, Pandas UDF improvements. >> In addition, this release continues to focus on usability, stability, and >> polish while resolving around 1100 tickets. >> >> We'd like to thank our contributors and users for their contributions and >> early feedback to this release. This release would not have been possible >> without you. >> >> To download Spark 2.4.0, head over to the download page: >> http://spark.apache.org/downloads.html >> >> To view the release notes: >> https://spark.apache.org/releases/spark-release-2-4-0.html >> >> Thanks, >> Wenchen >> >> PS: If you see any issues with the release notes, webpage or published >> artifacts, please contact me directly off-list. >> >
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
+user@ >> -- Forwarded message - >> From: Wenchen Fan >> Date: Thu, Nov 8, 2018 at 10:55 PM >> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >> To: Spark dev list >> >> >> Hi all, >> >> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds >> Barrier Execution Mode for better integration with deep learning frameworks, >> introduces 30+ built-in and higher-order functions to deal with complex data >> type easier, improves the K8s integration, along with experimental Scala >> 2.12 support. Other major updates include the built-in Avro data source, >> Image data source, flexible streaming sinks, elimination of the 2GB block >> size limitation during transfer, Pandas UDF improvements. In addition, this >> release continues to focus on usability, stability, and polish while >> resolving around 1100 tickets. >> >> We'd like to thank our contributors and users for their contributions and >> early feedback to this release. This release would not have been possible >> without you. >> >> To download Spark 2.4.0, head over to the download page: >> http://spark.apache.org/downloads.html >> >> To view the release notes: >> https://spark.apache.org/releases/spark-release-2-4-0.html >> >> Thanks, >> Wenchen >> >> PS: If you see any issues with the release notes, webpage or published >> artifacts, please contact me directly off-list. -- Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [ANNOUNCE] Announcing Apache Spark 2.4.0
resend On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan wrote: > > > -- Forwarded message - > From: Wenchen Fan > Date: Thu, Nov 8, 2018 at 10:55 PM > Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 > To: Spark dev list > > > Hi all, > > Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds > Barrier Execution Mode for better integration with deep learning > frameworks, introduces 30+ built-in and higher-order functions to deal with > complex data type easier, improves the K8s integration, along with > experimental Scala 2.12 support. Other major updates include the built-in > Avro data source, Image data source, flexible streaming sinks, elimination > of the 2GB block size limitation during transfer, Pandas UDF improvements. > In addition, this release continues to focus on usability, stability, and > polish while resolving around 1100 tickets. > > We'd like to thank our contributors and users for their contributions and > early feedback to this release. This release would not have been possible > without you. > > To download Spark 2.4.0, head over to the download page: > http://spark.apache.org/downloads.html > > To view the release notes: > https://spark.apache.org/releases/spark-release-2-4-0.html > > Thanks, > Wenchen > > PS: If you see any issues with the release notes, webpage or published > artifacts, please contact me directly off-list. >
