Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread Xiao Li 
+1 Thanks, Dongjoon!

Xiao



On Mon, May 17, 2021 at 8:45 PM Kent Yao  wrote:

> +1. thanks Dongjoon
>
> *Kent Yao *
> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> *a spark enthusiast*
> *kyuubi is a unified multi-tenant JDBC
> interface for large-scale data processing and analytics, built on top
> of Apache Spark .*
> *spark-authorizer A Spark
> SQL extension which provides SQL Standard Authorization for **Apache
> Spark .*
> *spark-postgres  A library for
> reading data from and transferring data to Postgres / Greenplum with Spark
> SQL and DataFrames, 10~100x faster.*
> *itatchi A** library t**hat
> brings useful functions from various modern database management systems to 
> **Apache
> Spark .*
>
>
>
> On 05/18/2021 10:57，John Zhuge 
> wrote：
>
> +1, thanks Dongjoon!
>
> On Mon, May 17, 2021 at 7:50 PM Yuming Wang  wrote:
>
>> +1.
>>
>> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon  wrote:
>>
>>> +1 thanks for driving me
>>>
>>> On Tue, 18 May 2021, 09:33 Holden Karau,  wrote:
>>>
 +1 and thanks for volunteering to be the RM :)

 On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro 
 wrote:

> Thank you, Dongjoon~ sgtm, too.
>
> On Tue, May 18, 2021 at 7:34 AM Cheng Su 
> wrote:
>
>> +1 for a new release, thanks Dongjoon!
>>
>> Cheng Su
>>
>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:
>>
>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>
>>
>> Liang-Chi
>>
>>
>> Dongjoon Hyun-2 wrote
>> > Hi, All.
>> >
>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>> > new 172 patches including 9 correctness patches and 4 K8s
>> patches arrived
>> > at branch-3.1.
>> >
>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>> release at
>> > 3.1 line?
>> > I'd like to volunteer for the release manager for Apache Spark
>> 3.1.2.
>> > I'm thinking about starting the first RC next week.
>> >
>> > $ git log --oneline v3.1.1..HEAD | wc -l
>> >  172
>> >
>> > # Known correctness issues
>> > SPARK-34534 New protocol FetchShuffleBlocks in
>> OneForOneBlockFetcher
>> > lead to data loss or correctness
>> > SPARK-34545 PySpark Python UDF return inconsistent results
>> when
>> > applying 2 UDFs with different return type to 2 columns together
>> > SPARK-34681 Full outer shuffled hash join when building
>> left side
>> > produces wrong result
>> > SPARK-34719 fail if the view query has duplicated column
>> names
>> > SPARK-34794 Nested higher-order functions broken in DSL
>> > SPARK-34829 transform_values return identical values when
>> it's used
>> > with udf that returns reference type
>> > SPARK-34833 Apply right-padding correctly for correlated
>> subqueries
>> > SPARK-35381 Fix lambda variable name issues in nested
>> DataFrame
>> > functions in R APIs
>> > SPARK-35382 Fix lambda variable name issues in nested
>> DataFrame
>> > functions in Python APIs
>> >
>> > # Notable K8s patches since K8s GA
>> > SPARK-34674Close SparkContext after the Main method has
>> finished
>> > SPARK-34948Add ownerReference to executor configmap to fix
>> leakages
>> > SPARK-34820add apt-update before gnupg install
>> > SPARK-34361In case of downscaling avoid killing of
>> executors already
>> > known by the scheduler backend in the pod allocator
>> >
>> > Bests,
>> > Dongjoon.
>>
>>
>>
>>
>>
>> --
>> Sent from:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>>
>
> --
> ---
> Takeshi Yamamuro
>
 --
 Twitter: https://twitter.com/holdenkarau
 Books (Learning Spark, High Performance Spark, etc.):
 https://amzn.to/2MaRAG9  
 YouTube Live Streams: https://www.youtube.com/user/holdenkarau

>>>
>
> --
> John Zhuge
>
>

--

Re: [ANNOUNCE] Apache Spark 2.4.8 released

2021-05-17 Thread Dongjoon Hyun 
Finally! Thank you, Liang-Chi.

Bests,
Dongjoon.


On Mon, May 17, 2021 at 10:14 PM Takeshi Yamamuro 
wrote:

> Thank you for the release work, Liang-Chi~
>
> On Tue, May 18, 2021 at 2:11 PM Hyukjin Kwon  wrote:
>
>> Yay!
>>
>> 2021년 5월 18일 (화) 오후 12:57, Liang-Chi Hsieh 님이 작성:
>>
>>> We are happy to announce the availability of Spark 2.4.8!
>>>
>>> Spark 2.4.8 is a maintenance release containing stability, correctness,
>>> and
>>> security fixes.
>>> This release is based on the branch-2.4 maintenance branch of Spark. We
>>> strongly recommend all 2.4 users to upgrade to this stable release.
>>>
>>> To download Spark 2.4.8, head over to the download page:
>>> http://spark.apache.org/downloads.html
>>>
>>> Note that you might need to clear your browser cache or to use
>>> `Private`/`Incognito` mode according to your browsers.
>>>
>>> To view the release notes:
>>> https://spark.apache.org/releases/spark-release-2-4-8.html
>>>
>>> We would like to acknowledge all community members for contributing to
>>> this
>>> release. This release would not have been possible without you.
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>
> --
> ---
> Takeshi Yamamuro
>

Re: [ANNOUNCE] Apache Spark 2.4.8 released

2021-05-17 Thread Takeshi Yamamuro 
Thank you for the release work, Liang-Chi~

On Tue, May 18, 2021 at 2:11 PM Hyukjin Kwon  wrote:

> Yay!
>
> 2021년 5월 18일 (화) 오후 12:57, Liang-Chi Hsieh 님이 작성:
>
>> We are happy to announce the availability of Spark 2.4.8!
>>
>> Spark 2.4.8 is a maintenance release containing stability, correctness,
>> and
>> security fixes.
>> This release is based on the branch-2.4 maintenance branch of Spark. We
>> strongly recommend all 2.4 users to upgrade to this stable release.
>>
>> To download Spark 2.4.8, head over to the download page:
>> http://spark.apache.org/downloads.html
>>
>> Note that you might need to clear your browser cache or to use
>> `Private`/`Incognito` mode according to your browsers.
>>
>> To view the release notes:
>> https://spark.apache.org/releases/spark-release-2-4-8.html
>>
>> We would like to acknowledge all community members for contributing to
>> this
>> release. This release would not have been possible without you.
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

-- 
---
Takeshi Yamamuro

Re: [ANNOUNCE] Apache Spark 2.4.8 released

2021-05-17 Thread Hyukjin Kwon 
Yay!

2021년 5월 18일 (화) 오후 12:57, Liang-Chi Hsieh 님이 작성:

> We are happy to announce the availability of Spark 2.4.8!
>
> Spark 2.4.8 is a maintenance release containing stability, correctness, and
> security fixes.
> This release is based on the branch-2.4 maintenance branch of Spark. We
> strongly recommend all 2.4 users to upgrade to this stable release.
>
> To download Spark 2.4.8, head over to the download page:
> http://spark.apache.org/downloads.html
>
> Note that you might need to clear your browser cache or to use
> `Private`/`Incognito` mode according to your browsers.
>
> To view the release notes:
> https://spark.apache.org/releases/spark-release-2-4-8.html
>
> We would like to acknowledge all community members for contributing to this
> release. This release would not have been possible without you.
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

[ANNOUNCE] Apache Spark 2.4.8 released

2021-05-17 Thread Liang-Chi Hsieh 
We are happy to announce the availability of Spark 2.4.8!

Spark 2.4.8 is a maintenance release containing stability, correctness, and
security fixes. 
This release is based on the branch-2.4 maintenance branch of Spark. We
strongly recommend all 2.4 users to upgrade to this stable release.

To download Spark 2.4.8, head over to the download page:
http://spark.apache.org/downloads.html

Note that you might need to clear your browser cache or to use
`Private`/`Incognito` mode according to your browsers.

To view the release notes:
https://spark.apache.org/releases/spark-release-2-4-8.html

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.




--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread Kent Yao 







+1. thanks Dongjoon






  





















Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark.spark-authorizerA Spark SQL extension which provides SQL Standard Authorization for Apache Spark.spark-postgres A library for reading data from and transferring data to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.itatchiA library that brings useful functions from various modern database management systems to Apache Spark.
















 


On 05/18/2021 10:57，John Zhuge wrote： 


+1, thanks Dongjoon!On Mon, May 17, 2021 at 7:50 PM Yuming Wang  wrote:+1.On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon  wrote:+1 thanks for driving meOn Tue, 18 May 2021, 09:33 Holden Karau,  wrote:+1 and thanks for volunteering to be the RM :)On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro  wrote:Thank you, Dongjoon~ sgtm, too.On Tue, May 18, 2021 at 7:34 AM Cheng Su  wrote:+1 for a new release, thanks Dongjoon!

Cheng Su

On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:

    +1 sounds good. Thanks Dongjoon for volunteering on this!


    Liang-Chi


    Dongjoon Hyun-2 wrote
    > Hi, All.
    > 
    > Since Apache Spark 3.1.1 tag creation (Feb 21),
    > new 172 patches including 9 correctness patches and 4 K8s patches arrived
    > at branch-3.1.
    > 
    > Shall we make a new release, Apache Spark 3.1.2, as the second release at
    > 3.1 line?
    > I'd like to volunteer for the release manager for Apache Spark 3.1.2.
    > I'm thinking about starting the first RC next week.
    > 
    > $ git log --oneline v3.1.1..HEAD | wc -l
    >      172
    > 
    > # Known correctness issues
    > SPARK-34534     New protocol FetchShuffleBlocks in OneForOneBlockFetcher
    > lead to data loss or correctness
    > SPARK-34545     PySpark Python UDF return inconsistent results when
    > applying 2 UDFs with different return type to 2 columns together
    > SPARK-34681     Full outer shuffled hash join when building left side
    > produces wrong result
    > SPARK-34719     fail if the view query has duplicated column names
    > SPARK-34794     Nested higher-order functions broken in DSL
    > SPARK-34829     transform_values return identical values when it's used
    > with udf that returns reference type
    > SPARK-34833     Apply right-padding correctly for correlated subqueries
    > SPARK-35381     Fix lambda variable name issues in nested DataFrame
    > functions in R APIs
    > SPARK-35382     Fix lambda variable name issues in nested DataFrame
    > functions in Python APIs
    > 
    > # Notable K8s patches since K8s GA
    > SPARK-34674    Close SparkContext after the Main method has finished
    > SPARK-34948    Add ownerReference to executor configmap to fix leakages
    > SPARK-34820    add apt-update before gnupg install
    > SPARK-34361    In case of downscaling avoid killing of executors already
    > known by the scheduler backend in the pod allocator
    > 
    > Bests,
    > Dongjoon.





    --
    Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ 

    -
    To unsubscribe e-mail: dev-unsubscr...@spark.apache.org


-- ---Takeshi Yamamuro
-- Twitter: https://twitter.com/holdenkarauBooks (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 YouTube Live Streams: https://www.youtube.com/user/holdenkarau


-- John Zhuge

Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread Chao Sun 
+1. Thanks Dongjoon for doing this!

On Mon, May 17, 2021 at 7:58 PM John Zhuge  wrote:

> +1, thanks Dongjoon!
>
> On Mon, May 17, 2021 at 7:50 PM Yuming Wang  wrote:
>
>> +1.
>>
>> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon  wrote:
>>
>>> +1 thanks for driving me
>>>
>>> On Tue, 18 May 2021, 09:33 Holden Karau,  wrote:
>>>
 +1 and thanks for volunteering to be the RM :)

 On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro 
 wrote:

> Thank you, Dongjoon~ sgtm, too.
>
> On Tue, May 18, 2021 at 7:34 AM Cheng Su 
> wrote:
>
>> +1 for a new release, thanks Dongjoon!
>>
>> Cheng Su
>>
>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:
>>
>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>
>>
>> Liang-Chi
>>
>>
>> Dongjoon Hyun-2 wrote
>> > Hi, All.
>> >
>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>> > new 172 patches including 9 correctness patches and 4 K8s
>> patches arrived
>> > at branch-3.1.
>> >
>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>> release at
>> > 3.1 line?
>> > I'd like to volunteer for the release manager for Apache Spark
>> 3.1.2.
>> > I'm thinking about starting the first RC next week.
>> >
>> > $ git log --oneline v3.1.1..HEAD | wc -l
>> >  172
>> >
>> > # Known correctness issues
>> > SPARK-34534 New protocol FetchShuffleBlocks in
>> OneForOneBlockFetcher
>> > lead to data loss or correctness
>> > SPARK-34545 PySpark Python UDF return inconsistent results
>> when
>> > applying 2 UDFs with different return type to 2 columns together
>> > SPARK-34681 Full outer shuffled hash join when building
>> left side
>> > produces wrong result
>> > SPARK-34719 fail if the view query has duplicated column
>> names
>> > SPARK-34794 Nested higher-order functions broken in DSL
>> > SPARK-34829 transform_values return identical values when
>> it's used
>> > with udf that returns reference type
>> > SPARK-34833 Apply right-padding correctly for correlated
>> subqueries
>> > SPARK-35381 Fix lambda variable name issues in nested
>> DataFrame
>> > functions in R APIs
>> > SPARK-35382 Fix lambda variable name issues in nested
>> DataFrame
>> > functions in Python APIs
>> >
>> > # Notable K8s patches since K8s GA
>> > SPARK-34674Close SparkContext after the Main method has
>> finished
>> > SPARK-34948Add ownerReference to executor configmap to fix
>> leakages
>> > SPARK-34820add apt-update before gnupg install
>> > SPARK-34361In case of downscaling avoid killing of
>> executors already
>> > known by the scheduler backend in the pod allocator
>> >
>> > Bests,
>> > Dongjoon.
>>
>>
>>
>>
>>
>> --
>> Sent from:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>>
>
> --
> ---
> Takeshi Yamamuro
>
 --
 Twitter: https://twitter.com/holdenkarau
 Books (Learning Spark, High Performance Spark, etc.):
 https://amzn.to/2MaRAG9  
 YouTube Live Streams: https://www.youtube.com/user/holdenkarau

>>>
>
> --
> John Zhuge
>

Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread John Zhuge 
+1, thanks Dongjoon!

On Mon, May 17, 2021 at 7:50 PM Yuming Wang  wrote:

> +1.
>
> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon  wrote:
>
>> +1 thanks for driving me
>>
>> On Tue, 18 May 2021, 09:33 Holden Karau,  wrote:
>>
>>> +1 and thanks for volunteering to be the RM :)
>>>
>>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro 
>>> wrote:
>>>
 Thank you, Dongjoon~ sgtm, too.

 On Tue, May 18, 2021 at 7:34 AM Cheng Su 
 wrote:

> +1 for a new release, thanks Dongjoon!
>
> Cheng Su
>
> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:
>
> +1 sounds good. Thanks Dongjoon for volunteering on this!
>
>
> Liang-Chi
>
>
> Dongjoon Hyun-2 wrote
> > Hi, All.
> >
> > Since Apache Spark 3.1.1 tag creation (Feb 21),
> > new 172 patches including 9 correctness patches and 4 K8s
> patches arrived
> > at branch-3.1.
> >
> > Shall we make a new release, Apache Spark 3.1.2, as the second
> release at
> > 3.1 line?
> > I'd like to volunteer for the release manager for Apache Spark
> 3.1.2.
> > I'm thinking about starting the first RC next week.
> >
> > $ git log --oneline v3.1.1..HEAD | wc -l
> >  172
> >
> > # Known correctness issues
> > SPARK-34534 New protocol FetchShuffleBlocks in
> OneForOneBlockFetcher
> > lead to data loss or correctness
> > SPARK-34545 PySpark Python UDF return inconsistent results
> when
> > applying 2 UDFs with different return type to 2 columns together
> > SPARK-34681 Full outer shuffled hash join when building left
> side
> > produces wrong result
> > SPARK-34719 fail if the view query has duplicated column
> names
> > SPARK-34794 Nested higher-order functions broken in DSL
> > SPARK-34829 transform_values return identical values when
> it's used
> > with udf that returns reference type
> > SPARK-34833 Apply right-padding correctly for correlated
> subqueries
> > SPARK-35381 Fix lambda variable name issues in nested
> DataFrame
> > functions in R APIs
> > SPARK-35382 Fix lambda variable name issues in nested
> DataFrame
> > functions in Python APIs
> >
> > # Notable K8s patches since K8s GA
> > SPARK-34674Close SparkContext after the Main method has
> finished
> > SPARK-34948Add ownerReference to executor configmap to fix
> leakages
> > SPARK-34820add apt-update before gnupg install
> > SPARK-34361In case of downscaling avoid killing of executors
> already
> > known by the scheduler backend in the pod allocator
> >
> > Bests,
> > Dongjoon.
>
>
>
>
>
> --
> Sent from:
> http://apache-spark-developers-list.1001551.n3.nabble.com/
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
>

 --
 ---
 Takeshi Yamamuro

>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

-- 
John Zhuge

Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread Yuming Wang 
+1.

On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon  wrote:

> +1 thanks for driving me
>
> On Tue, 18 May 2021, 09:33 Holden Karau,  wrote:
>
>> +1 and thanks for volunteering to be the RM :)
>>
>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro 
>> wrote:
>>
>>> Thank you, Dongjoon~ sgtm, too.
>>>
>>> On Tue, May 18, 2021 at 7:34 AM Cheng Su  wrote:
>>>
 +1 for a new release, thanks Dongjoon!

 Cheng Su

 On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:

 +1 sounds good. Thanks Dongjoon for volunteering on this!


 Liang-Chi


 Dongjoon Hyun-2 wrote
 > Hi, All.
 >
 > Since Apache Spark 3.1.1 tag creation (Feb 21),
 > new 172 patches including 9 correctness patches and 4 K8s patches
 arrived
 > at branch-3.1.
 >
 > Shall we make a new release, Apache Spark 3.1.2, as the second
 release at
 > 3.1 line?
 > I'd like to volunteer for the release manager for Apache Spark
 3.1.2.
 > I'm thinking about starting the first RC next week.
 >
 > $ git log --oneline v3.1.1..HEAD | wc -l
 >  172
 >
 > # Known correctness issues
 > SPARK-34534 New protocol FetchShuffleBlocks in
 OneForOneBlockFetcher
 > lead to data loss or correctness
 > SPARK-34545 PySpark Python UDF return inconsistent results
 when
 > applying 2 UDFs with different return type to 2 columns together
 > SPARK-34681 Full outer shuffled hash join when building left
 side
 > produces wrong result
 > SPARK-34719 fail if the view query has duplicated column names
 > SPARK-34794 Nested higher-order functions broken in DSL
 > SPARK-34829 transform_values return identical values when
 it's used
 > with udf that returns reference type
 > SPARK-34833 Apply right-padding correctly for correlated
 subqueries
 > SPARK-35381 Fix lambda variable name issues in nested
 DataFrame
 > functions in R APIs
 > SPARK-35382 Fix lambda variable name issues in nested
 DataFrame
 > functions in Python APIs
 >
 > # Notable K8s patches since K8s GA
 > SPARK-34674Close SparkContext after the Main method has
 finished
 > SPARK-34948Add ownerReference to executor configmap to fix
 leakages
 > SPARK-34820add apt-update before gnupg install
 > SPARK-34361In case of downscaling avoid killing of executors
 already
 > known by the scheduler backend in the pod allocator
 >
 > Bests,
 > Dongjoon.





 --
 Sent from:
 http://apache-spark-developers-list.1001551.n3.nabble.com/


 -
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org



>>>
>>> --
>>> ---
>>> Takeshi Yamamuro
>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread Hyukjin Kwon 
+1 thanks for driving me

On Tue, 18 May 2021, 09:33 Holden Karau,  wrote:

> +1 and thanks for volunteering to be the RM :)
>
> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro 
> wrote:
>
>> Thank you, Dongjoon~ sgtm, too.
>>
>> On Tue, May 18, 2021 at 7:34 AM Cheng Su  wrote:
>>
>>> +1 for a new release, thanks Dongjoon!
>>>
>>> Cheng Su
>>>
>>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:
>>>
>>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>>
>>>
>>> Liang-Chi
>>>
>>>
>>> Dongjoon Hyun-2 wrote
>>> > Hi, All.
>>> >
>>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>>> > new 172 patches including 9 correctness patches and 4 K8s patches
>>> arrived
>>> > at branch-3.1.
>>> >
>>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>>> release at
>>> > 3.1 line?
>>> > I'd like to volunteer for the release manager for Apache Spark
>>> 3.1.2.
>>> > I'm thinking about starting the first RC next week.
>>> >
>>> > $ git log --oneline v3.1.1..HEAD | wc -l
>>> >  172
>>> >
>>> > # Known correctness issues
>>> > SPARK-34534 New protocol FetchShuffleBlocks in
>>> OneForOneBlockFetcher
>>> > lead to data loss or correctness
>>> > SPARK-34545 PySpark Python UDF return inconsistent results when
>>> > applying 2 UDFs with different return type to 2 columns together
>>> > SPARK-34681 Full outer shuffled hash join when building left
>>> side
>>> > produces wrong result
>>> > SPARK-34719 fail if the view query has duplicated column names
>>> > SPARK-34794 Nested higher-order functions broken in DSL
>>> > SPARK-34829 transform_values return identical values when it's
>>> used
>>> > with udf that returns reference type
>>> > SPARK-34833 Apply right-padding correctly for correlated
>>> subqueries
>>> > SPARK-35381 Fix lambda variable name issues in nested DataFrame
>>> > functions in R APIs
>>> > SPARK-35382 Fix lambda variable name issues in nested DataFrame
>>> > functions in Python APIs
>>> >
>>> > # Notable K8s patches since K8s GA
>>> > SPARK-34674Close SparkContext after the Main method has
>>> finished
>>> > SPARK-34948Add ownerReference to executor configmap to fix
>>> leakages
>>> > SPARK-34820add apt-update before gnupg install
>>> > SPARK-34361In case of downscaling avoid killing of executors
>>> already
>>> > known by the scheduler backend in the pod allocator
>>> >
>>> > Bests,
>>> > Dongjoon.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from:
>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread Holden Karau 
+1 and thanks for volunteering to be the RM :)

On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro 
wrote:

> Thank you, Dongjoon~ sgtm, too.
>
> On Tue, May 18, 2021 at 7:34 AM Cheng Su  wrote:
>
>> +1 for a new release, thanks Dongjoon!
>>
>> Cheng Su
>>
>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:
>>
>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>
>>
>> Liang-Chi
>>
>>
>> Dongjoon Hyun-2 wrote
>> > Hi, All.
>> >
>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>> > new 172 patches including 9 correctness patches and 4 K8s patches
>> arrived
>> > at branch-3.1.
>> >
>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>> release at
>> > 3.1 line?
>> > I'd like to volunteer for the release manager for Apache Spark
>> 3.1.2.
>> > I'm thinking about starting the first RC next week.
>> >
>> > $ git log --oneline v3.1.1..HEAD | wc -l
>> >  172
>> >
>> > # Known correctness issues
>> > SPARK-34534 New protocol FetchShuffleBlocks in
>> OneForOneBlockFetcher
>> > lead to data loss or correctness
>> > SPARK-34545 PySpark Python UDF return inconsistent results when
>> > applying 2 UDFs with different return type to 2 columns together
>> > SPARK-34681 Full outer shuffled hash join when building left
>> side
>> > produces wrong result
>> > SPARK-34719 fail if the view query has duplicated column names
>> > SPARK-34794 Nested higher-order functions broken in DSL
>> > SPARK-34829 transform_values return identical values when it's
>> used
>> > with udf that returns reference type
>> > SPARK-34833 Apply right-padding correctly for correlated
>> subqueries
>> > SPARK-35381 Fix lambda variable name issues in nested DataFrame
>> > functions in R APIs
>> > SPARK-35382 Fix lambda variable name issues in nested DataFrame
>> > functions in Python APIs
>> >
>> > # Notable K8s patches since K8s GA
>> > SPARK-34674Close SparkContext after the Main method has finished
>> > SPARK-34948Add ownerReference to executor configmap to fix
>> leakages
>> > SPARK-34820add apt-update before gnupg install
>> > SPARK-34361In case of downscaling avoid killing of executors
>> already
>> > known by the scheduler backend in the pod allocator
>> >
>> > Bests,
>> > Dongjoon.
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>>
>
> --
> ---
> Takeshi Yamamuro
>
-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread Takeshi Yamamuro 
Thank you, Dongjoon~ sgtm, too.

On Tue, May 18, 2021 at 7:34 AM Cheng Su  wrote:

> +1 for a new release, thanks Dongjoon!
>
> Cheng Su
>
> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:
>
> +1 sounds good. Thanks Dongjoon for volunteering on this!
>
>
> Liang-Chi
>
>
> Dongjoon Hyun-2 wrote
> > Hi, All.
> >
> > Since Apache Spark 3.1.1 tag creation (Feb 21),
> > new 172 patches including 9 correctness patches and 4 K8s patches
> arrived
> > at branch-3.1.
> >
> > Shall we make a new release, Apache Spark 3.1.2, as the second
> release at
> > 3.1 line?
> > I'd like to volunteer for the release manager for Apache Spark 3.1.2.
> > I'm thinking about starting the first RC next week.
> >
> > $ git log --oneline v3.1.1..HEAD | wc -l
> >  172
> >
> > # Known correctness issues
> > SPARK-34534 New protocol FetchShuffleBlocks in
> OneForOneBlockFetcher
> > lead to data loss or correctness
> > SPARK-34545 PySpark Python UDF return inconsistent results when
> > applying 2 UDFs with different return type to 2 columns together
> > SPARK-34681 Full outer shuffled hash join when building left side
> > produces wrong result
> > SPARK-34719 fail if the view query has duplicated column names
> > SPARK-34794 Nested higher-order functions broken in DSL
> > SPARK-34829 transform_values return identical values when it's
> used
> > with udf that returns reference type
> > SPARK-34833 Apply right-padding correctly for correlated
> subqueries
> > SPARK-35381 Fix lambda variable name issues in nested DataFrame
> > functions in R APIs
> > SPARK-35382 Fix lambda variable name issues in nested DataFrame
> > functions in Python APIs
> >
> > # Notable K8s patches since K8s GA
> > SPARK-34674Close SparkContext after the Main method has finished
> > SPARK-34948Add ownerReference to executor configmap to fix
> leakages
> > SPARK-34820add apt-update before gnupg install
> > SPARK-34361In case of downscaling avoid killing of executors
> already
> > known by the scheduler backend in the pod allocator
> >
> > Bests,
> > Dongjoon.
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
>

-- 
---
Takeshi Yamamuro

Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread Cheng Su 
+1 for a new release, thanks Dongjoon!

Cheng Su

On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"  wrote:

+1 sounds good. Thanks Dongjoon for volunteering on this!


Liang-Chi


Dongjoon Hyun-2 wrote
> Hi, All.
> 
> Since Apache Spark 3.1.1 tag creation (Feb 21),
> new 172 patches including 9 correctness patches and 4 K8s patches arrived
> at branch-3.1.
> 
> Shall we make a new release, Apache Spark 3.1.2, as the second release at
> 3.1 line?
> I'd like to volunteer for the release manager for Apache Spark 3.1.2.
> I'm thinking about starting the first RC next week.
> 
> $ git log --oneline v3.1.1..HEAD | wc -l
>  172
> 
> # Known correctness issues
> SPARK-34534 New protocol FetchShuffleBlocks in OneForOneBlockFetcher
> lead to data loss or correctness
> SPARK-34545 PySpark Python UDF return inconsistent results when
> applying 2 UDFs with different return type to 2 columns together
> SPARK-34681 Full outer shuffled hash join when building left side
> produces wrong result
> SPARK-34719 fail if the view query has duplicated column names
> SPARK-34794 Nested higher-order functions broken in DSL
> SPARK-34829 transform_values return identical values when it's used
> with udf that returns reference type
> SPARK-34833 Apply right-padding correctly for correlated subqueries
> SPARK-35381 Fix lambda variable name issues in nested DataFrame
> functions in R APIs
> SPARK-35382 Fix lambda variable name issues in nested DataFrame
> functions in Python APIs
> 
> # Notable K8s patches since K8s GA
> SPARK-34674Close SparkContext after the Main method has finished
> SPARK-34948Add ownerReference to executor configmap to fix leakages
> SPARK-34820add apt-update before gnupg install
> SPARK-34361In case of downscaling avoid killing of executors already
> known by the scheduler backend in the pod allocator
> 
> Bests,
> Dongjoon.





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ 

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread Liang-Chi Hsieh 
+1 sounds good. Thanks Dongjoon for volunteering on this!


Liang-Chi


Dongjoon Hyun-2 wrote
> Hi, All.
> 
> Since Apache Spark 3.1.1 tag creation (Feb 21),
> new 172 patches including 9 correctness patches and 4 K8s patches arrived
> at branch-3.1.
> 
> Shall we make a new release, Apache Spark 3.1.2, as the second release at
> 3.1 line?
> I'd like to volunteer for the release manager for Apache Spark 3.1.2.
> I'm thinking about starting the first RC next week.
> 
> $ git log --oneline v3.1.1..HEAD | wc -l
>  172
> 
> # Known correctness issues
> SPARK-34534 New protocol FetchShuffleBlocks in OneForOneBlockFetcher
> lead to data loss or correctness
> SPARK-34545 PySpark Python UDF return inconsistent results when
> applying 2 UDFs with different return type to 2 columns together
> SPARK-34681 Full outer shuffled hash join when building left side
> produces wrong result
> SPARK-34719 fail if the view query has duplicated column names
> SPARK-34794 Nested higher-order functions broken in DSL
> SPARK-34829 transform_values return identical values when it's used
> with udf that returns reference type
> SPARK-34833 Apply right-padding correctly for correlated subqueries
> SPARK-35381 Fix lambda variable name issues in nested DataFrame
> functions in R APIs
> SPARK-35382 Fix lambda variable name issues in nested DataFrame
> functions in Python APIs
> 
> # Notable K8s patches since K8s GA
> SPARK-34674Close SparkContext after the Main method has finished
> SPARK-34948Add ownerReference to executor configmap to fix leakages
> SPARK-34820add apt-update before gnupg install
> SPARK-34361In case of downscaling avoid killing of executors already
> known by the scheduler backend in the pod allocator
> 
> Bests,
> Dongjoon.





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Apache Spark 3.1.2 Release?

2021-05-17 Thread Dongjoon Hyun 
Hi, All.

Since Apache Spark 3.1.1 tag creation (Feb 21),
new 172 patches including 9 correctness patches and 4 K8s patches arrived
at branch-3.1.

Shall we make a new release, Apache Spark 3.1.2, as the second release at
3.1 line?
I'd like to volunteer for the release manager for Apache Spark 3.1.2.
I'm thinking about starting the first RC next week.

$ git log --oneline v3.1.1..HEAD | wc -l
 172

# Known correctness issues
SPARK-34534 New protocol FetchShuffleBlocks in OneForOneBlockFetcher
lead to data loss or correctness
SPARK-34545 PySpark Python UDF return inconsistent results when
applying 2 UDFs with different return type to 2 columns together
SPARK-34681 Full outer shuffled hash join when building left side
produces wrong result
SPARK-34719 fail if the view query has duplicated column names
SPARK-34794 Nested higher-order functions broken in DSL
SPARK-34829 transform_values return identical values when it's used
with udf that returns reference type
SPARK-34833 Apply right-padding correctly for correlated subqueries
SPARK-35381 Fix lambda variable name issues in nested DataFrame
functions in R APIs
SPARK-35382 Fix lambda variable name issues in nested DataFrame
functions in Python APIs

# Notable K8s patches since K8s GA
SPARK-34674Close SparkContext after the Main method has finished
SPARK-34948Add ownerReference to executor configmap to fix leakages
SPARK-34820add apt-update before gnupg install
SPARK-34361In case of downscaling avoid killing of executors already
known by the scheduler backend in the pod allocator

Bests,
Dongjoon.