Unsubscribe

2021-05-18 Thread Dhruv Kumar 
Unsubscribe

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [ANNOUNCE] Apache Spark 2.4.8 released

2021-05-18 Thread Gengliang Wang 
Great work, Liang-Chi!

> On May 19, 2021, at 10:23 AM, 郑瑞峰  wrote:
> 
> Thank you for the great work, Liang-Chi!
> 
> 
> -- 原始邮件 --
> 发件人: "Chao Sun" ;
> 发送时间: 2021年5月19日(星期三) 凌晨1:28
> 收件人: "Maxim Gekk";
> 抄送: "Yuming Wang";"Jungtaek 
> Lim";"Wenchen 
> Fan";"Dongjoon Hyun";"Takeshi 
> Yamamuro";"Liang-Chi 
> Hsieh";"dev";
> 主题: Re: [ANNOUNCE] Apache Spark 2.4.8 released
> 
> Great work Liang-Chi!
> 
> On Tue, May 18, 2021 at 1:14 AM Maxim Gekk  > wrote:
> Congratulations everyone with the new release, and thanks to Liang-Chi.
> 
> Maxim Gekk
> Software Engineer
> Databricks, Inc.
> 
> 
> On Tue, May 18, 2021 at 11:06 AM Yuming Wang  > wrote:
> Great work, Liang-Chi!
> 
> On Tue, May 18, 2021 at 3:57 PM Jungtaek Lim  > wrote:
> Thanks for the huge efforts on driving the release!
> 
> On Tue, May 18, 2021 at 4:53 PM Wenchen Fan  > wrote:
> Thank you, Liang-Chi!
> 
> On Tue, May 18, 2021 at 1:32 PM Dongjoon Hyun  > wrote:
> Finally! Thank you, Liang-Chi.
> 
> Bests,
> Dongjoon.
> 
> 
> On Mon, May 17, 2021 at 10:14 PM Takeshi Yamamuro  > wrote:
> Thank you for the release work, Liang-Chi~
> 
> On Tue, May 18, 2021 at 2:11 PM Hyukjin Kwon  > wrote:
> Yay!
> 
> 2021년 5월 18일 (화) 오후 12:57, Liang-Chi Hsieh  >님이 작성:
> We are happy to announce the availability of Spark 2.4.8!
> 
> Spark 2.4.8 is a maintenance release containing stability, correctness, and
> security fixes. 
> This release is based on the branch-2.4 maintenance branch of Spark. We
> strongly recommend all 2.4 users to upgrade to this stable release.
> 
> To download Spark 2.4.8, head over to the download page:
> http://spark.apache.org/downloads.html 
> 
> 
> Note that you might need to clear your browser cache or to use
> `Private`/`Incognito` mode according to your browsers.
> 
> To view the release notes:
> https://spark.apache.org/releases/spark-release-2-4-8.html 
> 
> 
> We would like to acknowledge all community members for contributing to this
> release. This release would not have been possible without you.
> 
> 
> 
> 
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ 
> 
> 
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org 
> 
> 
> 
> 
> -- 
> ---
> Takeshi Yamamuro

Re: Apache Spark 3.1.2 Release?

2021-05-18 Thread Gengliang Wang 
Late +1, thank you, Dongjoon!

> On May 19, 2021, at 10:47 AM, Jungtaek Lim  
> wrote:
> 
> Late +1 here as well, thanks for volunteering!
> 
> 2021년 5월 19일 (수) 오전 11:24, 郑瑞峰  >님이 작성:
> late +1. thanks Dongjoon!
> 
> 
> -- 原始邮件 --
> 发件人: "Dongjoon Hyun"  >;
> 发送时间: 2021年5月19日(星期三) 凌晨1:29
> 收件人: "Wenchen Fan"mailto:cloud0...@gmail.com>>;
> 抄送: "Xiao Li"mailto:lix...@databricks.com>>;"Kent 
> Yao"mailto:yaooq...@gmail.com>>;"John 
> Zhuge"mailto:jzh...@apache.org>>;"Hyukjin 
> Kwon"mailto:gurwls...@gmail.com>>;"Holden 
> Karau"mailto:hol...@pigscanfly.ca>>;"Takeshi 
> Yamamuro" >;"dev" >;"Yuming Wang" >;
> 主题: Re: Apache Spark 3.1.2 Release?
> 
> Thank you all! I'll start to prepare.
> 
> Bests,
> Dongjoon.
> 
> On Tue, May 18, 2021 at 12:53 AM Wenchen Fan  > wrote:
> +1, thanks!
> 
> On Tue, May 18, 2021 at 1:37 PM Xiao Li  > wrote:
> +1 Thanks, Dongjoon!
> 
> Xiao
> 
> 
> 
> On Mon, May 17, 2021 at 8:45 PM Kent Yao  > wrote:
> +1. thanks Dongjoon
> 
> Kent Yao 
> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> a spark enthusiast
> kyuubi is a unified multi-tenant JDBC 
> interface for large-scale data processing and analytics, built on top of 
> Apache Spark .
> spark-authorizer A Spark SQL 
> extension which provides SQL Standard Authorization for Apache Spark 
> .
> spark-postgres  A library for 
> reading data from and transferring data to Postgres / Greenplum with Spark 
> SQL and DataFrames, 10~100x faster.
> itatchi A library that brings 
> useful functions from various modern database management systems to Apache 
> Spark .
> 
> 
>  
> 
> On 05/18/2021 10:57，John Zhuge  
> wrote：
> +1, thanks Dongjoon!
> 
> On Mon, May 17, 2021 at 7:50 PM Yuming Wang  > wrote:
> +1.
> 
> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon  > wrote:
> +1 thanks for driving me
> 
> On Tue, 18 May 2021, 09:33 Holden Karau,  > wrote:
> +1 and thanks for volunteering to be the RM :)
> 
> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro  > wrote:
> Thank you, Dongjoon~ sgtm, too.
> 
> On Tue, May 18, 2021 at 7:34 AM Cheng Su  wrote:
> +1 for a new release, thanks Dongjoon!
> 
> Cheng Su
> 
> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  > wrote:
> 
> +1 sounds good. Thanks Dongjoon for volunteering on this!
> 
> 
> Liang-Chi
> 
> 
> Dongjoon Hyun-2 wrote
> > Hi, All.
> > 
> > Since Apache Spark 3.1.1 tag creation (Feb 21),
> > new 172 patches including 9 correctness patches and 4 K8s patches 
> arrived
> > at branch-3.1.
> > 
> > Shall we make a new release, Apache Spark 3.1.2, as the second release 
> at
> > 3.1 line?
> > I'd like to volunteer for the release manager for Apache Spark 3.1.2.
> > I'm thinking about starting the first RC next week.
> > 
> > $ git log --oneline v3.1.1..HEAD | wc -l
> >  172
> > 
> > # Known correctness issues
> > SPARK-34534 New protocol FetchShuffleBlocks in OneForOneBlockFetcher
> > lead to data loss or correctness
> > SPARK-34545 PySpark Python UDF return inconsistent results when
> > applying 2 UDFs with different return type to 2 columns together
> > SPARK-34681 Full outer shuffled hash join when building left side
> > produces wrong result
> > SPARK-34719 fail if the view query has duplicated column names
> > SPARK-34794 Nested higher-order functions broken in DSL
> > SPARK-34829 transform_values return identical values when it's used
> > with udf that returns reference type
> > SPARK-34833 Apply right-padding correctly for correlated subqueries
> > SPARK-35381 Fix lambda variable name issues in nested DataFrame
> > functions in R APIs
> > SPARK-35382 Fix lambda variable name issues in nested DataFrame
> > functions in Python APIs
> > 
> > # Notable K8s patches since K8s GA
> > SPARK-34674Close SparkContext after the Main method has finished
> > SPARK-34948Add ownerReference to executor configmap to fix leakages
> > SPARK-34820add apt-update before gnupg install
> > SPARK-34361In case of downscaling avoid killing of executors already
> > known by the scheduler backend in the pod allocator
> > 
> > Bests,
> > Dongjoon.
> 
> 
> 
> 
> 
> --
> Sent from:

Re: Apache Spark 3.1.2 Release?

2021-05-18 Thread Jungtaek Lim 
Late +1 here as well, thanks for volunteering!

2021년 5월 19일 (수) 오전 11:24, 郑瑞峰 님이 작성:

> late +1. thanks Dongjoon!
>
>
> -- 原始邮件 --
> *发件人:* "Dongjoon Hyun" ;
> *发送时间:* 2021年5月19日(星期三) 凌晨1:29
> *收件人:* "Wenchen Fan";
> *抄送:* "Xiao Li";"Kent Yao";"John
> Zhuge";"Hyukjin Kwon";"Holden
> Karau";"Takeshi Yamamuro" >;"dev";"Yuming Wang";
> *主题:* Re: Apache Spark 3.1.2 Release?
>
> Thank you all! I'll start to prepare.
>
> Bests,
> Dongjoon.
>
> On Tue, May 18, 2021 at 12:53 AM Wenchen Fan  wrote:
>
>> +1, thanks!
>>
>> On Tue, May 18, 2021 at 1:37 PM Xiao Li  wrote:
>>
>>> +1 Thanks, Dongjoon!
>>>
>>> Xiao
>>>
>>>
>>>
>>> On Mon, May 17, 2021 at 8:45 PM Kent Yao  wrote:
>>>
 +1. thanks Dongjoon

 *Kent Yao *
 @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
 *a spark enthusiast*
 *kyuubi is a
 unified multi-tenant JDBC interface for large-scale data processing and
 analytics, built on top of Apache Spark .*
 *spark-authorizer A Spark
 SQL extension which provides SQL Standard Authorization for **Apache
 Spark .*
 *spark-postgres  A library
 for reading data from and transferring data to Postgres / Greenplum with
 Spark SQL and DataFrames, 10~100x faster.*
 *itatchi A** library t**hat
 brings useful functions from various modern database management systems to 
 **Apache
 Spark .*



 On 05/18/2021 10:57，John Zhuge 
 wrote：

 +1, thanks Dongjoon!

 On Mon, May 17, 2021 at 7:50 PM Yuming Wang  wrote:

> +1.
>
> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon 
> wrote:
>
>> +1 thanks for driving me
>>
>> On Tue, 18 May 2021, 09:33 Holden Karau, 
>> wrote:
>>
>>> +1 and thanks for volunteering to be the RM :)
>>>
>>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <
>>> linguin@gmail.com> wrote:
>>>
 Thank you, Dongjoon~ sgtm, too.

 On Tue, May 18, 2021 at 7:34 AM Cheng Su 
 wrote:

> +1 for a new release, thanks Dongjoon!
>
> Cheng Su
>
> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:
>
> +1 sounds good. Thanks Dongjoon for volunteering on this!
>
>
> Liang-Chi
>
>
> Dongjoon Hyun-2 wrote
> > Hi, All.
> >
> > Since Apache Spark 3.1.1 tag creation (Feb 21),
> > new 172 patches including 9 correctness patches and 4 K8s
> patches arrived
> > at branch-3.1.
> >
> > Shall we make a new release, Apache Spark 3.1.2, as the
> second release at
> > 3.1 line?
> > I'd like to volunteer for the release manager for Apache
> Spark 3.1.2.
> > I'm thinking about starting the first RC next week.
> >
> > $ git log --oneline v3.1.1..HEAD | wc -l
> >  172
> >
> > # Known correctness issues
> > SPARK-34534 New protocol FetchShuffleBlocks in
> OneForOneBlockFetcher
> > lead to data loss or correctness
> > SPARK-34545 PySpark Python UDF return inconsistent
> results when
> > applying 2 UDFs with different return type to 2 columns
> together
> > SPARK-34681 Full outer shuffled hash join when building
> left side
> > produces wrong result
> > SPARK-34719 fail if the view query has duplicated column
> names
> > SPARK-34794 Nested higher-order functions broken in DSL
> > SPARK-34829 transform_values return identical values
> when it's used
> > with udf that returns reference type
> > SPARK-34833 Apply right-padding correctly for correlated
> subqueries
> > SPARK-35381 Fix lambda variable name issues in nested
> DataFrame
> > functions in R APIs
> > SPARK-35382 Fix lambda variable name issues in nested
> DataFrame
> > functions in Python APIs
> >
> > # Notable K8s patches since K8s GA
> > SPARK-34674Close SparkContext after the Main method has
> finished
> > SPARK-34948Add ownerReference to executor configmap to
> fix leakages
> > SPARK-34820add apt-update before gnupg install
> > SPARK-34361In case of downscaling avoid killing of
> executors already
> > known by the scheduler backend in the pod allocator
>

WholeStageCodeGen + DSv2

2021-05-18 Thread Andrew Melo 
Hello,

When reading a very wide (> 1000 cols) input, WholeStageCodeGen blows
past the 64kB source limit and fails. Looking at the generated code, a
big part of the code is simply the DSv2 convention that the codegen'd
variable names are the same as the columns instead of something more
compact like 'c1', 'c2', etc..

Would there be any interest in accepting a patch that shortens these
variable names to try and stay under the limit?

Thanks
Andrew

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

回复： Apache Spark 3.1.2 Release?

2021-05-18 Thread 郑瑞峰 
late +1. thanks Dongjoon!



--原始邮件--
发件人:
"Dongjoon Hyun" 
   
http://apache-spark-developers-list.1001551.n3.nabble.com/ 
 
   
-
   To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
 
 
 



-- 
---
Takeshi Yamamuro



 

-- 
Twitter:https://twitter.com/holdenkarau

Books (Learning Spark, High Performance Spark, 
etc.):https://amzn.to/2MaRAG9;
YouTube Live Streams:https://www.youtube.com/user/holdenkarau









 
 
 



-- 
John Zhuge

  
 
 



--

回复： [ANNOUNCE] Apache Spark 2.4.8 released

2021-05-18 Thread 郑瑞峰 
Thank you for the great work, Liang-Chi!



--原始邮件--
发件人:
"Chao Sun"  
  
http://spark.apache.org/downloads.html
 
 Note that you might need to clear your browser cache or to use
 `Private`/`Incognito` mode according to your browsers.
 
 To view the release notes:
 https://spark.apache.org/releases/spark-release-2-4-8.html
 
 We would like to acknowledge all community members for contributing to this
 release. This release would not have been possible without you.
 
 
 
 
 --
 Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
 
 -
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
 
 
 



-- 
---
Takeshi Yamamuro

Secrets store for DSv2

2021-05-18 Thread Andrew Melo 
Hello,

When implementing a DSv2 datasource, where is an appropriate place to
store/transmit secrets from the driver to the executors? Is there
built-in spark functionality for that, or is my best bet to stash it
as a member variable in one of the classes that gets sent to the
executors?

Thanks!
Andrew

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Apache Spark 3.1.2 Release?

2021-05-18 Thread Dongjoon Hyun 
Thank you all! I'll start to prepare.

Bests,
Dongjoon.

On Tue, May 18, 2021 at 12:53 AM Wenchen Fan  wrote:

> +1, thanks!
>
> On Tue, May 18, 2021 at 1:37 PM Xiao Li  wrote:
>
>> +1 Thanks, Dongjoon!
>>
>> Xiao
>>
>>
>>
>> On Mon, May 17, 2021 at 8:45 PM Kent Yao  wrote:
>>
>>> +1. thanks Dongjoon
>>>
>>> *Kent Yao *
>>> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
>>> *a spark enthusiast*
>>> *kyuubi is a
>>> unified multi-tenant JDBC interface for large-scale data processing and
>>> analytics, built on top of Apache Spark .*
>>> *spark-authorizer A Spark
>>> SQL extension which provides SQL Standard Authorization for **Apache
>>> Spark .*
>>> *spark-postgres  A library
>>> for reading data from and transferring data to Postgres / Greenplum with
>>> Spark SQL and DataFrames, 10~100x faster.*
>>> *itatchi A** library t**hat
>>> brings useful functions from various modern database management systems to 
>>> **Apache
>>> Spark .*
>>>
>>>
>>>
>>> On 05/18/2021 10:57，John Zhuge 
>>> wrote：
>>>
>>> +1, thanks Dongjoon!
>>>
>>> On Mon, May 17, 2021 at 7:50 PM Yuming Wang  wrote:
>>>
 +1.

 On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon 
 wrote:

> +1 thanks for driving me
>
> On Tue, 18 May 2021, 09:33 Holden Karau,  wrote:
>
>> +1 and thanks for volunteering to be the RM :)
>>
>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <
>> linguin@gmail.com> wrote:
>>
>>> Thank you, Dongjoon~ sgtm, too.
>>>
>>> On Tue, May 18, 2021 at 7:34 AM Cheng Su 
>>> wrote:
>>>
 +1 for a new release, thanks Dongjoon!

 Cheng Su

 On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:

 +1 sounds good. Thanks Dongjoon for volunteering on this!


 Liang-Chi


 Dongjoon Hyun-2 wrote
 > Hi, All.
 >
 > Since Apache Spark 3.1.1 tag creation (Feb 21),
 > new 172 patches including 9 correctness patches and 4 K8s
 patches arrived
 > at branch-3.1.
 >
 > Shall we make a new release, Apache Spark 3.1.2, as the
 second release at
 > 3.1 line?
 > I'd like to volunteer for the release manager for Apache
 Spark 3.1.2.
 > I'm thinking about starting the first RC next week.
 >
 > $ git log --oneline v3.1.1..HEAD | wc -l
 >  172
 >
 > # Known correctness issues
 > SPARK-34534 New protocol FetchShuffleBlocks in
 OneForOneBlockFetcher
 > lead to data loss or correctness
 > SPARK-34545 PySpark Python UDF return inconsistent
 results when
 > applying 2 UDFs with different return type to 2 columns
 together
 > SPARK-34681 Full outer shuffled hash join when building
 left side
 > produces wrong result
 > SPARK-34719 fail if the view query has duplicated column
 names
 > SPARK-34794 Nested higher-order functions broken in DSL
 > SPARK-34829 transform_values return identical values when
 it's used
 > with udf that returns reference type
 > SPARK-34833 Apply right-padding correctly for correlated
 subqueries
 > SPARK-35381 Fix lambda variable name issues in nested
 DataFrame
 > functions in R APIs
 > SPARK-35382 Fix lambda variable name issues in nested
 DataFrame
 > functions in Python APIs
 >
 > # Notable K8s patches since K8s GA
 > SPARK-34674Close SparkContext after the Main method has
 finished
 > SPARK-34948Add ownerReference to executor configmap to
 fix leakages
 > SPARK-34820add apt-update before gnupg install
 > SPARK-34361In case of downscaling avoid killing of
 executors already
 > known by the scheduler backend in the pod allocator
 >
 > Bests,
 > Dongjoon.





 --
 Sent from:
 http://apache-spark-developers-list.1001551.n3.nabble.com/


 -
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org



>>>
>>> --
>>> ---
>>> Takeshi Yamamuro
>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark,

Re: [ANNOUNCE] Apache Spark 2.4.8 released

2021-05-18 Thread Chao Sun 
Great work Liang-Chi!

On Tue, May 18, 2021 at 1:14 AM Maxim Gekk 
wrote:

> Congratulations everyone with the new release, and thanks to Liang-Chi.
>
> Maxim Gekk
>
> Software Engineer
>
> Databricks, Inc.
>
>
> On Tue, May 18, 2021 at 11:06 AM Yuming Wang  wrote:
>
>> Great work, Liang-Chi!
>>
>> On Tue, May 18, 2021 at 3:57 PM Jungtaek Lim <
>> kabhwan.opensou...@gmail.com> wrote:
>>
>>> Thanks for the huge efforts on driving the release!
>>>
>>> On Tue, May 18, 2021 at 4:53 PM Wenchen Fan  wrote:
>>>
 Thank you, Liang-Chi!

 On Tue, May 18, 2021 at 1:32 PM Dongjoon Hyun 
 wrote:

> Finally! Thank you, Liang-Chi.
>
> Bests,
> Dongjoon.
>
>
> On Mon, May 17, 2021 at 10:14 PM Takeshi Yamamuro <
> linguin@gmail.com> wrote:
>
>> Thank you for the release work, Liang-Chi~
>>
>> On Tue, May 18, 2021 at 2:11 PM Hyukjin Kwon 
>> wrote:
>>
>>> Yay!
>>>
>>> 2021년 5월 18일 (화) 오후 12:57, Liang-Chi Hsieh 님이 작성:
>>>
 We are happy to announce the availability of Spark 2.4.8!

 Spark 2.4.8 is a maintenance release containing stability,
 correctness, and
 security fixes.
 This release is based on the branch-2.4 maintenance branch of
 Spark. We
 strongly recommend all 2.4 users to upgrade to this stable release.

 To download Spark 2.4.8, head over to the download page:
 http://spark.apache.org/downloads.html

 Note that you might need to clear your browser cache or to use
 `Private`/`Incognito` mode according to your browsers.

 To view the release notes:
 https://spark.apache.org/releases/spark-release-2-4-8.html

 We would like to acknowledge all community members for contributing
 to this
 release. This release would not have been possible without you.




 --
 Sent from:
 http://apache-spark-developers-list.1001551.n3.nabble.com/


 -
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org


>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>

Re: [ANNOUNCE] Apache Spark 2.4.8 released

2021-05-18 Thread Maxim Gekk 
Congratulations everyone with the new release, and thanks to Liang-Chi.

Maxim Gekk

Software Engineer

Databricks, Inc.


On Tue, May 18, 2021 at 11:06 AM Yuming Wang  wrote:

> Great work, Liang-Chi!
>
> On Tue, May 18, 2021 at 3:57 PM Jungtaek Lim 
> wrote:
>
>> Thanks for the huge efforts on driving the release!
>>
>> On Tue, May 18, 2021 at 4:53 PM Wenchen Fan  wrote:
>>
>>> Thank you, Liang-Chi!
>>>
>>> On Tue, May 18, 2021 at 1:32 PM Dongjoon Hyun 
>>> wrote:
>>>
 Finally! Thank you, Liang-Chi.

 Bests,
 Dongjoon.


 On Mon, May 17, 2021 at 10:14 PM Takeshi Yamamuro <
 linguin@gmail.com> wrote:

> Thank you for the release work, Liang-Chi~
>
> On Tue, May 18, 2021 at 2:11 PM Hyukjin Kwon 
> wrote:
>
>> Yay!
>>
>> 2021년 5월 18일 (화) 오후 12:57, Liang-Chi Hsieh 님이 작성:
>>
>>> We are happy to announce the availability of Spark 2.4.8!
>>>
>>> Spark 2.4.8 is a maintenance release containing stability,
>>> correctness, and
>>> security fixes.
>>> This release is based on the branch-2.4 maintenance branch of Spark.
>>> We
>>> strongly recommend all 2.4 users to upgrade to this stable release.
>>>
>>> To download Spark 2.4.8, head over to the download page:
>>> http://spark.apache.org/downloads.html
>>>
>>> Note that you might need to clear your browser cache or to use
>>> `Private`/`Incognito` mode according to your browsers.
>>>
>>> To view the release notes:
>>> https://spark.apache.org/releases/spark-release-2-4-8.html
>>>
>>> We would like to acknowledge all community members for contributing
>>> to this
>>> release. This release would not have been possible without you.
>>>
>>>
>>>
>>>
>>> --
>>> Sent from:
>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>
> --
> ---
> Takeshi Yamamuro
>

Re: [ANNOUNCE] Apache Spark 2.4.8 released

2021-05-18 Thread Yuming Wang 
Great work, Liang-Chi!

On Tue, May 18, 2021 at 3:57 PM Jungtaek Lim 
wrote:

> Thanks for the huge efforts on driving the release!
>
> On Tue, May 18, 2021 at 4:53 PM Wenchen Fan  wrote:
>
>> Thank you, Liang-Chi!
>>
>> On Tue, May 18, 2021 at 1:32 PM Dongjoon Hyun 
>> wrote:
>>
>>> Finally! Thank you, Liang-Chi.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Mon, May 17, 2021 at 10:14 PM Takeshi Yamamuro 
>>> wrote:
>>>
 Thank you for the release work, Liang-Chi~

 On Tue, May 18, 2021 at 2:11 PM Hyukjin Kwon 
 wrote:

> Yay!
>
> 2021년 5월 18일 (화) 오후 12:57, Liang-Chi Hsieh 님이 작성:
>
>> We are happy to announce the availability of Spark 2.4.8!
>>
>> Spark 2.4.8 is a maintenance release containing stability,
>> correctness, and
>> security fixes.
>> This release is based on the branch-2.4 maintenance branch of Spark.
>> We
>> strongly recommend all 2.4 users to upgrade to this stable release.
>>
>> To download Spark 2.4.8, head over to the download page:
>> http://spark.apache.org/downloads.html
>>
>> Note that you might need to clear your browser cache or to use
>> `Private`/`Incognito` mode according to your browsers.
>>
>> To view the release notes:
>> https://spark.apache.org/releases/spark-release-2-4-8.html
>>
>> We would like to acknowledge all community members for contributing
>> to this
>> release. This release would not have been possible without you.
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

 --
 ---
 Takeshi Yamamuro

>>>

Re: [ANNOUNCE] Apache Spark 2.4.8 released

2021-05-18 Thread Jungtaek Lim 
Thanks for the huge efforts on driving the release!

On Tue, May 18, 2021 at 4:53 PM Wenchen Fan  wrote:

> Thank you, Liang-Chi!
>
> On Tue, May 18, 2021 at 1:32 PM Dongjoon Hyun 
> wrote:
>
>> Finally! Thank you, Liang-Chi.
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Mon, May 17, 2021 at 10:14 PM Takeshi Yamamuro 
>> wrote:
>>
>>> Thank you for the release work, Liang-Chi~
>>>
>>> On Tue, May 18, 2021 at 2:11 PM Hyukjin Kwon 
>>> wrote:
>>>
 Yay!

 2021년 5월 18일 (화) 오후 12:57, Liang-Chi Hsieh 님이 작성:

> We are happy to announce the availability of Spark 2.4.8!
>
> Spark 2.4.8 is a maintenance release containing stability,
> correctness, and
> security fixes.
> This release is based on the branch-2.4 maintenance branch of Spark. We
> strongly recommend all 2.4 users to upgrade to this stable release.
>
> To download Spark 2.4.8, head over to the download page:
> http://spark.apache.org/downloads.html
>
> Note that you might need to clear your browser cache or to use
> `Private`/`Incognito` mode according to your browsers.
>
> To view the release notes:
> https://spark.apache.org/releases/spark-release-2-4-8.html
>
> We would like to acknowledge all community members for contributing to
> this
> release. This release would not have been possible without you.
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
>>>
>>> --
>>> ---
>>> Takeshi Yamamuro
>>>
>>

Re: [ANNOUNCE] Apache Spark 2.4.8 released

2021-05-18 Thread Wenchen Fan 
Thank you, Liang-Chi!

On Tue, May 18, 2021 at 1:32 PM Dongjoon Hyun 
wrote:

> Finally! Thank you, Liang-Chi.
>
> Bests,
> Dongjoon.
>
>
> On Mon, May 17, 2021 at 10:14 PM Takeshi Yamamuro 
> wrote:
>
>> Thank you for the release work, Liang-Chi~
>>
>> On Tue, May 18, 2021 at 2:11 PM Hyukjin Kwon  wrote:
>>
>>> Yay!
>>>
>>> 2021년 5월 18일 (화) 오후 12:57, Liang-Chi Hsieh 님이 작성:
>>>
 We are happy to announce the availability of Spark 2.4.8!

 Spark 2.4.8 is a maintenance release containing stability, correctness,
 and
 security fixes.
 This release is based on the branch-2.4 maintenance branch of Spark. We
 strongly recommend all 2.4 users to upgrade to this stable release.

 To download Spark 2.4.8, head over to the download page:
 http://spark.apache.org/downloads.html

 Note that you might need to clear your browser cache or to use
 `Private`/`Incognito` mode according to your browsers.

 To view the release notes:
 https://spark.apache.org/releases/spark-release-2-4-8.html

 We would like to acknowledge all community members for contributing to
 this
 release. This release would not have been possible without you.




 --
 Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

 -
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org


>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>

Re: Apache Spark 3.1.2 Release?

2021-05-18 Thread Wenchen Fan 
+1, thanks!

On Tue, May 18, 2021 at 1:37 PM Xiao Li  wrote:

> +1 Thanks, Dongjoon!
>
> Xiao
>
>
>
> On Mon, May 17, 2021 at 8:45 PM Kent Yao  wrote:
>
>> +1. thanks Dongjoon
>>
>> *Kent Yao *
>> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
>> *a spark enthusiast*
>> *kyuubi is a
>> unified multi-tenant JDBC interface for large-scale data processing and
>> analytics, built on top of Apache Spark .*
>> *spark-authorizer A Spark
>> SQL extension which provides SQL Standard Authorization for **Apache
>> Spark .*
>> *spark-postgres  A library
>> for reading data from and transferring data to Postgres / Greenplum with
>> Spark SQL and DataFrames, 10~100x faster.*
>> *itatchi A** library t**hat
>> brings useful functions from various modern database management systems to 
>> **Apache
>> Spark .*
>>
>>
>>
>> On 05/18/2021 10:57，John Zhuge 
>> wrote：
>>
>> +1, thanks Dongjoon!
>>
>> On Mon, May 17, 2021 at 7:50 PM Yuming Wang  wrote:
>>
>>> +1.
>>>
>>> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon 
>>> wrote:
>>>
 +1 thanks for driving me

 On Tue, 18 May 2021, 09:33 Holden Karau,  wrote:

> +1 and thanks for volunteering to be the RM :)
>
> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <
> linguin@gmail.com> wrote:
>
>> Thank you, Dongjoon~ sgtm, too.
>>
>> On Tue, May 18, 2021 at 7:34 AM Cheng Su 
>> wrote:
>>
>>> +1 for a new release, thanks Dongjoon!
>>>
>>> Cheng Su
>>>
>>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:
>>>
>>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>>
>>>
>>> Liang-Chi
>>>
>>>
>>> Dongjoon Hyun-2 wrote
>>> > Hi, All.
>>> >
>>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>>> > new 172 patches including 9 correctness patches and 4 K8s
>>> patches arrived
>>> > at branch-3.1.
>>> >
>>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>>> release at
>>> > 3.1 line?
>>> > I'd like to volunteer for the release manager for Apache Spark
>>> 3.1.2.
>>> > I'm thinking about starting the first RC next week.
>>> >
>>> > $ git log --oneline v3.1.1..HEAD | wc -l
>>> >  172
>>> >
>>> > # Known correctness issues
>>> > SPARK-34534 New protocol FetchShuffleBlocks in
>>> OneForOneBlockFetcher
>>> > lead to data loss or correctness
>>> > SPARK-34545 PySpark Python UDF return inconsistent results
>>> when
>>> > applying 2 UDFs with different return type to 2 columns
>>> together
>>> > SPARK-34681 Full outer shuffled hash join when building
>>> left side
>>> > produces wrong result
>>> > SPARK-34719 fail if the view query has duplicated column
>>> names
>>> > SPARK-34794 Nested higher-order functions broken in DSL
>>> > SPARK-34829 transform_values return identical values when
>>> it's used
>>> > with udf that returns reference type
>>> > SPARK-34833 Apply right-padding correctly for correlated
>>> subqueries
>>> > SPARK-35381 Fix lambda variable name issues in nested
>>> DataFrame
>>> > functions in R APIs
>>> > SPARK-35382 Fix lambda variable name issues in nested
>>> DataFrame
>>> > functions in Python APIs
>>> >
>>> > # Notable K8s patches since K8s GA
>>> > SPARK-34674Close SparkContext after the Main method has
>>> finished
>>> > SPARK-34948Add ownerReference to executor configmap to fix
>>> leakages
>>> > SPARK-34820add apt-update before gnupg install
>>> > SPARK-34361In case of downscaling avoid killing of
>>> executors already
>>> > known by the scheduler backend in the pod allocator
>>> >
>>> > Bests,
>>> > Dongjoon.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from:
>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

>>
>> --
>> John Zhuge
>>
>>
>
> --
>
>