Re: [VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Ruifeng Zheng 
+1

On Thu, Jun 22, 2023 at 1:11 PM Dongjoon Hyun 
wrote:

> +1
>
> Dongjoon
>
> On Wed, Jun 21, 2023 at 8:56 PM Hyukjin Kwon  wrote:
>
>> +1
>>
>> On Thu, 22 Jun 2023 at 02:20, Jacek Laskowski  wrote:
>>
>>> +0
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> 
>>> "The Internals Of" Online Books 
>>> Follow me on https://twitter.com/jaceklaskowski
>>>
>>> 
>>>
>>>
>>> On Wed, Jun 21, 2023 at 5:11 PM Amanda Liu 
>>> wrote:
>>>
 Hi all,

 I'd like to start the vote for SPIP: PySpark Test Framework.

 The high-level summary for the SPIP is that it proposes an official
 test framework for PySpark. Currently, there are only disparate open-source
 repos and blog posts for PySpark testing resources. We can streamline and
 simplify the testing process by incorporating test features, such as a
 PySpark Test Base class (which allows tests to share Spark sessions) and
 test util functions (for example, asserting dataframe and schema equality).

 *SPIP doc:*
 https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v

 *JIRA ticket:* https://issues.apache.org/jira/browse/SPARK-44042

 *Discussion thread:*
 https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n

 Please vote on the SPIP for the next 72 hours:
 [ ] +1: Accept the proposal as an official SPIP
 [ ] +0
 [ ] -1: I don’t think this is a good idea because __.

 Thank you!

 Best,
 Amanda Liu

>>>

Re: [VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Dongjoon Hyun 
+1

Dongjoon

On Wed, Jun 21, 2023 at 8:56 PM Hyukjin Kwon  wrote:

> +1
>
> On Thu, 22 Jun 2023 at 02:20, Jacek Laskowski  wrote:
>
>> +0
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> "The Internals Of" Online Books 
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> 
>>
>>
>> On Wed, Jun 21, 2023 at 5:11 PM Amanda Liu 
>> wrote:
>>
>>> Hi all,
>>>
>>> I'd like to start the vote for SPIP: PySpark Test Framework.
>>>
>>> The high-level summary for the SPIP is that it proposes an official test
>>> framework for PySpark. Currently, there are only disparate open-source
>>> repos and blog posts for PySpark testing resources. We can streamline and
>>> simplify the testing process by incorporating test features, such as a
>>> PySpark Test Base class (which allows tests to share Spark sessions) and
>>> test util functions (for example, asserting dataframe and schema equality).
>>>
>>> *SPIP doc:*
>>> https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
>>>
>>> *JIRA ticket:* https://issues.apache.org/jira/browse/SPARK-44042
>>>
>>> *Discussion thread:*
>>> https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n
>>>
>>> Please vote on the SPIP for the next 72 hours:
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don’t think this is a good idea because __.
>>>
>>> Thank you!
>>>
>>> Best,
>>> Amanda Liu
>>>
>>

Re: [VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Hyukjin Kwon 
+1

On Thu, 22 Jun 2023 at 02:20, Jacek Laskowski  wrote:

> +0
>
> Pozdrawiam,
> Jacek Laskowski
> 
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>
>
> On Wed, Jun 21, 2023 at 5:11 PM Amanda Liu 
> wrote:
>
>> Hi all,
>>
>> I'd like to start the vote for SPIP: PySpark Test Framework.
>>
>> The high-level summary for the SPIP is that it proposes an official test
>> framework for PySpark. Currently, there are only disparate open-source
>> repos and blog posts for PySpark testing resources. We can streamline and
>> simplify the testing process by incorporating test features, such as a
>> PySpark Test Base class (which allows tests to share Spark sessions) and
>> test util functions (for example, asserting dataframe and schema equality).
>>
>> *SPIP doc:*
>> https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
>>
>> *JIRA ticket:* https://issues.apache.org/jira/browse/SPARK-44042
>>
>> *Discussion thread:*
>> https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n
>>
>> Please vote on the SPIP for the next 72 hours:
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because __.
>>
>> Thank you!
>>
>> Best,
>> Amanda Liu
>>
>

Re: [VOTE] Release Spark 3.4.1 (RC1)

2023-06-21 Thread Xinrong Meng 
+1

Thank you!

On Wed, Jun 21, 2023 at 1:14 AM Peter Toth  wrote:

> +1
>
> Ruifeng Zheng  ezt írta (időpont: 2023. jún. 21.,
> Sze, 9:43):
>
>> +1
>>
>> On Wed, Jun 21, 2023 at 2:26 PM huaxin gao 
>> wrote:
>>
>>> +1
>>>
>>> On Tue, Jun 20, 2023 at 11:21 PM Hyukjin Kwon 
>>> wrote:
>>>
 +1

 On Wed, 21 Jun 2023 at 14:23, yangjie01  wrote:

> +1
>
>
> 在 2023/6/21 13:20，“L. C. Hsieh” vii...@gmail.com>> 写入:
>
>
> +1
>
>
> On Tue, Jun 20, 2023 at 8:48 PM Dongjoon Hyun  > wrote:
> >
> > +1
> >
> > Dongjoon
> >
> > On 2023/06/20 02:51:32 Jia Fan wrote:
> > > +1
> > >
> > > Dongjoon Hyun mailto:dongj...@apache.org>>
> 于2023年6月20日周二 10:41写道：
> > >
> > > > Please vote on releasing the following candidate as Apache Spark
> version
> > > > 3.4.1.
> > > >
> > > > The vote is open until June 23rd 1AM (PST) and passes if a
> majority +1 PMC
> > > > votes are cast, with a minimum of 3 +1 votes.
> > > >
> > > > [ ] +1 Release this package as Apache Spark 3.4.1
> > > > [ ] -1 Do not release this package because ...
> > > >
> > > > To learn more about Apache Spark, please see
> https://spark.apache.org/ 
> > > >
> > > > The tag to be voted on is v3.4.1-rc1 (commit
> > > > 6b1ff22dde1ead51cbf370be6e48a802daae58b6)
> > > > https://github.com/apache/spark/tree/v3.4.1-rc1 <
> https://github.com/apache/spark/tree/v3.4.1-rc1>
> > > >
> > > > The release files, including signatures, digests, etc. can be
> found at:
> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/ <
> https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/>
> > > >
> > > > Signatures used for Spark RCs can be found in this file:
> > > > https://dist.apache.org/repos/dist/dev/spark/KEYS <
> https://dist.apache.org/repos/dist/dev/spark/KEYS>
> > > >
> > > > The staging repository for this release can be found at:
> > > >
> https://repository.apache.org/content/repositories/orgapachespark-1443/
> <
> https://repository.apache.org/content/repositories/orgapachespark-1443/
> >
> > > >
> > > > The documentation corresponding to this release can be found at:
> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/ <
> https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/>
> > > >
> > > > The list of bug fixes going into 3.4.1 can be found at the
> following URL:
> > > > https://issues.apache.org/jira/projects/SPARK/versions/12352874
> 
> > > >
> > > > This release is using the release script of the tag v3.4.1-rc1.
> > > >
> > > > FAQ
> > > >
> > > > =
> > > > How can I help test this release?
> > > > =
> > > >
> > > > If you are a Spark user, you can help us test this release by
> taking
> > > > an existing Spark workload and running on this release
> candidate, then
> > > > reporting any regressions.
> > > >
> > > > If you're working in PySpark you can set up a virtual env and
> install
> > > > the current RC and see if anything important breaks, in the
> Java/Scala
> > > > you can add the staging repository to your projects resolvers
> and test
> > > > with the RC (make sure to clean up the artifact cache
> before/after so
> > > > you don't end up building with a out of date RC going forward).
> > > >
> > > > ===
> > > > What should happen to JIRA tickets still targeting 3.4.1?
> > > > ===
> > > >
> > > > The current list of open tickets targeted at 3.4.1 can be found
> at:
> > > > https://issues.apache.org/jira/projects/SPARK <
> https://issues.apache.org/jira/projects/SPARK> and search for "Target
> > > > Version/s" = 3.4.1
> > > >
> > > > Committers should look at those and triage. Extremely important
> bug
> > > > fixes, documentation, and API tweaks that impact compatibility
> should
> > > > be worked on immediately. Everything else please retarget to an
> > > > appropriate release.
> > > >
> > > > ==
> > > > But my bug isn't fixed?
> > > > ==
> > > >
> > > > In order to make timely releases, we will typically not hold the
> > > > release unless the bug in question is a regression from the
> previous
> > > > release. That being said, if there is something which is a
> regression
> > > > that has not been correctly targeted please ping me or a
> committer to
> > > > help target the issue.
> > > >
> > >
> >
> >

Re: [VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Jacek Laskowski 
+0

Pozdrawiam,
Jacek Laskowski

"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Wed, Jun 21, 2023 at 5:11 PM Amanda Liu 
wrote:

> Hi all,
>
> I'd like to start the vote for SPIP: PySpark Test Framework.
>
> The high-level summary for the SPIP is that it proposes an official test
> framework for PySpark. Currently, there are only disparate open-source
> repos and blog posts for PySpark testing resources. We can streamline and
> simplify the testing process by incorporating test features, such as a
> PySpark Test Base class (which allows tests to share Spark sessions) and
> test util functions (for example, asserting dataframe and schema equality).
>
> *SPIP doc:*
> https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
>
> *JIRA ticket:* https://issues.apache.org/jira/browse/SPARK-44042
>
> *Discussion thread:*
> https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n
>
> Please vote on the SPIP for the next 72 hours:
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don’t think this is a good idea because __.
>
> Thank you!
>
> Best,
> Amanda Liu
>

Re: [VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Amanda Liu 
Yes, let's extend the vote by two days in light of traveling for pride
weekend and conferences.

Best,
Amanda Liu


On Wed, Jun 21, 2023 at 8:41 AM Maciej  wrote:

> +1
>
> --
> Best regards,
> Maciej Szymkiewicz
>
> Web: https://zero323.net
> PGP: A30CEF0C31A501EC
>
>
> On 6/21/23 17:35, Holden Karau wrote:
>
> A small request, it’s pride weekend in San Francisco where some of the
> core developers are and right before one of the larger spark related
> conferences so more folks might be traveling than normal. Could we maybe
> extend the vote out an extra day or two just to give folks a chance to be
> heard?
>
> On Wed, Jun 21, 2023 at 8:30 AM Reynold Xin  wrote:
>
>> +1
>>
>> This is a great idea.
>>
>>
>> On Wed, Jun 21, 2023 at 8:29 AM, Holden Karau 
>> wrote:
>>
>>> I’d like to start with a +1, better Python testing tools integrated into
>>> the project make sense.
>>>
>>> On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu 
>>> wrote:
>>>
 Hi all,

 I'd like to start the vote for SPIP: PySpark Test Framework.

 The high-level summary for the SPIP is that it proposes an official
 test framework for PySpark. Currently, there are only disparate open-source
 repos and blog posts for PySpark testing resources. We can streamline and
 simplify the testing process by incorporating test features, such as a
 PySpark Test Base class (which allows tests to share Spark sessions) and
 test util functions (for example, asserting dataframe and schema equality).

 *SPIP doc:*
 https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v

 *JIRA ticket:* https://issues.apache.org/jira/browse/SPARK-44042

 *Discussion thread:*
 https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n

 Please vote on the SPIP for the next 72 hours:
 [ ] +1: Accept the proposal as an official SPIP
 [ ] +0
 [ ] -1: I don’t think this is a good idea because __.

 Thank you!

 Best,
 Amanda Liu

>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
>
>

Re: [VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Maciej 

+1

--
Best regards,
Maciej Szymkiewicz

Web:https://zero323.net
PGP: A30CEF0C31A501EC


On 6/21/23 17:35, Holden Karau wrote:
A small request, it’s pride weekend in San Francisco where some of the 
core developers are and right before one of the larger spark related 
conferences so more folks might be traveling than normal. Could we 
maybe extend the vote out an extra day or two just to give folks a 
chance to be heard?


On Wed, Jun 21, 2023 at 8:30 AM Reynold Xin  wrote:

+1

This is a great idea.


On Wed, Jun 21, 2023 at 8:29 AM, Holden Karau
 wrote:

I’d like to start with a +1, better Python testing tools
integrated into the project make sense.

On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu
 wrote:

Hi all,

I'd like to start the vote for SPIP: PySpark Test Framework.

The high-level summary for the SPIP is that it proposes an
official test framework for PySpark. Currently, there are
only disparate open-source repos and blog posts for
PySpark testing resources. We can streamline and simplify
the testing process by incorporating test features, such
as a PySpark Test Base class (which allows tests to share
Spark sessions) and test util functions (for example,
asserting dataframe and schema equality).

*SPIP doc:*

https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v

*JIRA ticket:*
https://issues.apache.org/jira/browse/SPARK-44042

*Discussion thread:*
https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n

Please vote on the SPIP for the next 72 hours:
[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because __.

Thank you!

Best,
Amanda Liu

-- 
Twitter: https://twitter.com/holdenkarau

Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 
YouTube Live Streams: https://www.youtube.com/user/holdenkarau


--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.): 
https://amzn.to/2MaRAG9 

YouTube Live Streams: https://www.youtube.com/user/holdenkarau




OpenPGP_signature
Description: OpenPGP digital signature

Re: [VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Holden Karau 
A small request, it’s pride weekend in San Francisco where some of the core
developers are and right before one of the larger spark related conferences
so more folks might be traveling than normal. Could we maybe extend the
vote out an extra day or two just to give folks a chance to be heard?

On Wed, Jun 21, 2023 at 8:30 AM Reynold Xin  wrote:

> +1
>
> This is a great idea.
>
>
> On Wed, Jun 21, 2023 at 8:29 AM, Holden Karau 
> wrote:
>
>> I’d like to start with a +1, better Python testing tools integrated into
>> the project make sense.
>>
>> On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu 
>> wrote:
>>
>>> Hi all,
>>>
>>> I'd like to start the vote for SPIP: PySpark Test Framework.
>>>
>>> The high-level summary for the SPIP is that it proposes an official test
>>> framework for PySpark. Currently, there are only disparate open-source
>>> repos and blog posts for PySpark testing resources. We can streamline and
>>> simplify the testing process by incorporating test features, such as a
>>> PySpark Test Base class (which allows tests to share Spark sessions) and
>>> test util functions (for example, asserting dataframe and schema equality).
>>>
>>> *SPIP doc:*
>>> https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
>>>
>>> *JIRA ticket:* https://issues.apache.org/jira/browse/SPARK-44042
>>>
>>> *Discussion thread:*
>>> https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n
>>>
>>> Please vote on the SPIP for the next 72 hours:
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don’t think this is a good idea because __.
>>>
>>> Thank you!
>>>
>>> Best,
>>> Amanda Liu
>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: [VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Mich Talebzadeh 
+1 for me

Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Wed, 21 Jun 2023 at 16:30, Holden Karau  wrote:

> I’d like to start with a +1, better Python testing tools integrated into
> the project make sense.
>
> On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu 
> wrote:
>
>> Hi all,
>>
>> I'd like to start the vote for SPIP: PySpark Test Framework.
>>
>> The high-level summary for the SPIP is that it proposes an official test
>> framework for PySpark. Currently, there are only disparate open-source
>> repos and blog posts for PySpark testing resources. We can streamline and
>> simplify the testing process by incorporating test features, such as a
>> PySpark Test Base class (which allows tests to share Spark sessions) and
>> test util functions (for example, asserting dataframe and schema equality).
>>
>> *SPIP doc:*
>> https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
>>
>> *JIRA ticket:* https://issues.apache.org/jira/browse/SPARK-44042
>>
>> *Discussion thread:*
>> https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n
>>
>> Please vote on the SPIP for the next 72 hours:
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because __.
>>
>> Thank you!
>>
>> Best,
>> Amanda Liu
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: [VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Reynold Xin 
+1

This is a great idea.

On Wed, Jun 21, 2023 at 8:29 AM, Holden Karau < hol...@pigscanfly.ca > wrote:

> 
> I’d like to start with a +1, better Python testing tools integrated into
> the project make sense.
> 
> On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu < amandastephanieliu@ gmail. com
> ( amandastephanie...@gmail.com ) > wrote:
> 
> 
>> Hi all,
>> 
>> I'd like to start the vote for SPIP: PySpark Test Framework.
>> 
>> The high-level summary for the SPIP is that it proposes an official test
>> framework for PySpark. Currently, there are only disparate open-source
>> repos and blog posts for PySpark testing resources. We can streamline and
>> simplify the testing process by incorporating test features, such as a
>> PySpark Test Base class (which allows tests to share Spark sessions) and
>> test util functions (for example, asserting dataframe and schema
>> equality).
>> 
>> *SPIP doc:* https:/ / docs. google. com/ document/ d/ 
>> 1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/
>> edit#heading=h. f5f0u2riv07v (
>> https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
>> )
>> 
>> 
>> *JIRA ticket:* https:/ / issues. apache. org/ jira/ browse/ SPARK-44042 (
>> https://issues.apache.org/jira/browse/SPARK-44042 )
>> 
>> *Discussion thread:* https:/ / lists. apache. org/ thread/ 
>> trwgbgn3ycoj8b8k8lkxko2hql23o41n
>> ( https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n )
>> 
>> Please vote on the SPIP for the next 72 hours:
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because __.
>> 
>> Thank you!
>> 
>> Best,
>> Amanda Liu
>> 
>> 
> 
> --
> Twitter: https:/ / twitter. com/ holdenkarau (
> https://twitter.com/holdenkarau )
> 
> Books (Learning Spark, High Performance Spark, etc.): https:/ / amzn. to/ 
> 2MaRAG9
> ( https://amzn.to/2MaRAG9 )
> YouTube Live Streams: https:/ / www. youtube. com/ user/ holdenkarau (
> https://www.youtube.com/user/holdenkarau )
>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: [VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Holden Karau 
I’d like to start with a +1, better Python testing tools integrated into
the project make sense.

On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu 
wrote:

> Hi all,
>
> I'd like to start the vote for SPIP: PySpark Test Framework.
>
> The high-level summary for the SPIP is that it proposes an official test
> framework for PySpark. Currently, there are only disparate open-source
> repos and blog posts for PySpark testing resources. We can streamline and
> simplify the testing process by incorporating test features, such as a
> PySpark Test Base class (which allows tests to share Spark sessions) and
> test util functions (for example, asserting dataframe and schema equality).
>
> *SPIP doc:*
> https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
>
> *JIRA ticket:* https://issues.apache.org/jira/browse/SPARK-44042
>
> *Discussion thread:*
> https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n
>
> Please vote on the SPIP for the next 72 hours:
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don’t think this is a good idea because __.
>
> Thank you!
>
> Best,
> Amanda Liu
>
-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

[VOTE][SPIP] PySpark Test Framework

2023-06-21 Thread Amanda Liu 
Hi all,

I'd like to start the vote for SPIP: PySpark Test Framework.

The high-level summary for the SPIP is that it proposes an official test
framework for PySpark. Currently, there are only disparate open-source
repos and blog posts for PySpark testing resources. We can streamline and
simplify the testing process by incorporating test features, such as a
PySpark Test Base class (which allows tests to share Spark sessions) and
test util functions (for example, asserting dataframe and schema equality).

*SPIP doc:*
https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v

*JIRA ticket:* https://issues.apache.org/jira/browse/SPARK-44042

*Discussion thread:*
https://lists.apache.org/thread/trwgbgn3ycoj8b8k8lkxko2hql23o41n

Please vote on the SPIP for the next 72 hours:
[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because __.

Thank you!

Best,
Amanda Liu

Unsubscribe

2023-06-21 Thread Gary Liu 
Unsubscribe

Re: [VOTE] Apache Spark PMC asks Databricks to differentiate its Spark version string

2023-06-21 Thread Steve Loughran 
I'd say everyone should *and* http UA in all the clients who make calls of
object stores should, as it helps field issues there. s3a and abfs clients
do provide the ability to add params there -please set them in your
deployments

On Fri, 16 Jun 2023 at 21:53, Dongjoon Hyun  wrote:

> Please vote on the following statement. The vote is open until June 23th
> 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of
> 3 +1 votes.
>
> Apache Spark PMC asks Databricks to differentiate its Spark
> version string to avoid confusions because Apache Spark PMC
> is responsible for ensuring to follow ASF requirements[1] and
> respects ASF's legal advice [2, 3],
>
> [ ] +1 Yes
> [ ] -1 No because ...
>
> 
> 1. https://www.apache.org/foundation/governance/pmcs#organization
> 2. https://lists.apache.org/thread/mzhggd0rpz8t4d7vdsbhkp38mvd3lty4
> 3. https://www.apache.org/foundation/marks/downstream.html#source
>

Re: [VOTE] Release Spark 3.4.1 (RC1)

2023-06-21 Thread Peter Toth 
+1

Ruifeng Zheng  ezt írta (időpont: 2023. jún. 21., Sze,
9:43):

> +1
>
> On Wed, Jun 21, 2023 at 2:26 PM huaxin gao  wrote:
>
>> +1
>>
>> On Tue, Jun 20, 2023 at 11:21 PM Hyukjin Kwon 
>> wrote:
>>
>>> +1
>>>
>>> On Wed, 21 Jun 2023 at 14:23, yangjie01  wrote:
>>>
 +1


 在 2023/6/21 13:20，“L. C. Hsieh”>>> vii...@gmail.com>> 写入:


 +1


 On Tue, Jun 20, 2023 at 8:48 PM Dongjoon Hyun >>> > wrote:
 >
 > +1
 >
 > Dongjoon
 >
 > On 2023/06/20 02:51:32 Jia Fan wrote:
 > > +1
 > >
 > > Dongjoon Hyun mailto:dongj...@apache.org>>
 于2023年6月20日周二 10:41写道：
 > >
 > > > Please vote on releasing the following candidate as Apache Spark
 version
 > > > 3.4.1.
 > > >
 > > > The vote is open until June 23rd 1AM (PST) and passes if a
 majority +1 PMC
 > > > votes are cast, with a minimum of 3 +1 votes.
 > > >
 > > > [ ] +1 Release this package as Apache Spark 3.4.1
 > > > [ ] -1 Do not release this package because ...
 > > >
 > > > To learn more about Apache Spark, please see
 https://spark.apache.org/ 
 > > >
 > > > The tag to be voted on is v3.4.1-rc1 (commit
 > > > 6b1ff22dde1ead51cbf370be6e48a802daae58b6)
 > > > https://github.com/apache/spark/tree/v3.4.1-rc1 <
 https://github.com/apache/spark/tree/v3.4.1-rc1>
 > > >
 > > > The release files, including signatures, digests, etc. can be
 found at:
 > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/ <
 https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/>
 > > >
 > > > Signatures used for Spark RCs can be found in this file:
 > > > https://dist.apache.org/repos/dist/dev/spark/KEYS <
 https://dist.apache.org/repos/dist/dev/spark/KEYS>
 > > >
 > > > The staging repository for this release can be found at:
 > > >
 https://repository.apache.org/content/repositories/orgapachespark-1443/
 <
 https://repository.apache.org/content/repositories/orgapachespark-1443/
 >
 > > >
 > > > The documentation corresponding to this release can be found at:
 > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/ <
 https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/>
 > > >
 > > > The list of bug fixes going into 3.4.1 can be found at the
 following URL:
 > > > https://issues.apache.org/jira/projects/SPARK/versions/12352874 <
 https://issues.apache.org/jira/projects/SPARK/versions/12352874>
 > > >
 > > > This release is using the release script of the tag v3.4.1-rc1.
 > > >
 > > > FAQ
 > > >
 > > > =
 > > > How can I help test this release?
 > > > =
 > > >
 > > > If you are a Spark user, you can help us test this release by
 taking
 > > > an existing Spark workload and running on this release candidate,
 then
 > > > reporting any regressions.
 > > >
 > > > If you're working in PySpark you can set up a virtual env and
 install
 > > > the current RC and see if anything important breaks, in the
 Java/Scala
 > > > you can add the staging repository to your projects resolvers and
 test
 > > > with the RC (make sure to clean up the artifact cache
 before/after so
 > > > you don't end up building with a out of date RC going forward).
 > > >
 > > > ===
 > > > What should happen to JIRA tickets still targeting 3.4.1?
 > > > ===
 > > >
 > > > The current list of open tickets targeted at 3.4.1 can be found
 at:
 > > > https://issues.apache.org/jira/projects/SPARK <
 https://issues.apache.org/jira/projects/SPARK> and search for "Target
 > > > Version/s" = 3.4.1
 > > >
 > > > Committers should look at those and triage. Extremely important
 bug
 > > > fixes, documentation, and API tweaks that impact compatibility
 should
 > > > be worked on immediately. Everything else please retarget to an
 > > > appropriate release.
 > > >
 > > > ==
 > > > But my bug isn't fixed?
 > > > ==
 > > >
 > > > In order to make timely releases, we will typically not hold the
 > > > release unless the bug in question is a regression from the
 previous
 > > > release. That being said, if there is something which is a
 regression
 > > > that has not been correctly targeted please ping me or a
 committer to
 > > > help target the issue.
 > > >
 > >
 >
 > -
 > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> dev-unsubscr...@spark.apache.org>
 >

Re: [VOTE] Release Spark 3.4.1 (RC1)

2023-06-21 Thread Ruifeng Zheng 
+1

On Wed, Jun 21, 2023 at 2:26 PM huaxin gao  wrote:

> +1
>
> On Tue, Jun 20, 2023 at 11:21 PM Hyukjin Kwon 
> wrote:
>
>> +1
>>
>> On Wed, 21 Jun 2023 at 14:23, yangjie01  wrote:
>>
>>> +1
>>>
>>>
>>> 在 2023/6/21 13:20，“L. C. Hsieh”>> vii...@gmail.com>> 写入:
>>>
>>>
>>> +1
>>>
>>>
>>> On Tue, Jun 20, 2023 at 8:48 PM Dongjoon Hyun >> > wrote:
>>> >
>>> > +1
>>> >
>>> > Dongjoon
>>> >
>>> > On 2023/06/20 02:51:32 Jia Fan wrote:
>>> > > +1
>>> > >
>>> > > Dongjoon Hyun mailto:dongj...@apache.org>>
>>> 于2023年6月20日周二 10:41写道：
>>> > >
>>> > > > Please vote on releasing the following candidate as Apache Spark
>>> version
>>> > > > 3.4.1.
>>> > > >
>>> > > > The vote is open until June 23rd 1AM (PST) and passes if a
>>> majority +1 PMC
>>> > > > votes are cast, with a minimum of 3 +1 votes.
>>> > > >
>>> > > > [ ] +1 Release this package as Apache Spark 3.4.1
>>> > > > [ ] -1 Do not release this package because ...
>>> > > >
>>> > > > To learn more about Apache Spark, please see
>>> https://spark.apache.org/ 
>>> > > >
>>> > > > The tag to be voted on is v3.4.1-rc1 (commit
>>> > > > 6b1ff22dde1ead51cbf370be6e48a802daae58b6)
>>> > > > https://github.com/apache/spark/tree/v3.4.1-rc1 <
>>> https://github.com/apache/spark/tree/v3.4.1-rc1>
>>> > > >
>>> > > > The release files, including signatures, digests, etc. can be
>>> found at:
>>> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/ <
>>> https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/>
>>> > > >
>>> > > > Signatures used for Spark RCs can be found in this file:
>>> > > > https://dist.apache.org/repos/dist/dev/spark/KEYS <
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS>
>>> > > >
>>> > > > The staging repository for this release can be found at:
>>> > > >
>>> https://repository.apache.org/content/repositories/orgapachespark-1443/
>>> >> >
>>> > > >
>>> > > > The documentation corresponding to this release can be found at:
>>> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/ <
>>> https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/>
>>> > > >
>>> > > > The list of bug fixes going into 3.4.1 can be found at the
>>> following URL:
>>> > > > https://issues.apache.org/jira/projects/SPARK/versions/12352874 <
>>> https://issues.apache.org/jira/projects/SPARK/versions/12352874>
>>> > > >
>>> > > > This release is using the release script of the tag v3.4.1-rc1.
>>> > > >
>>> > > > FAQ
>>> > > >
>>> > > > =
>>> > > > How can I help test this release?
>>> > > > =
>>> > > >
>>> > > > If you are a Spark user, you can help us test this release by
>>> taking
>>> > > > an existing Spark workload and running on this release candidate,
>>> then
>>> > > > reporting any regressions.
>>> > > >
>>> > > > If you're working in PySpark you can set up a virtual env and
>>> install
>>> > > > the current RC and see if anything important breaks, in the
>>> Java/Scala
>>> > > > you can add the staging repository to your projects resolvers and
>>> test
>>> > > > with the RC (make sure to clean up the artifact cache before/after
>>> so
>>> > > > you don't end up building with a out of date RC going forward).
>>> > > >
>>> > > > ===
>>> > > > What should happen to JIRA tickets still targeting 3.4.1?
>>> > > > ===
>>> > > >
>>> > > > The current list of open tickets targeted at 3.4.1 can be found at:
>>> > > > https://issues.apache.org/jira/projects/SPARK <
>>> https://issues.apache.org/jira/projects/SPARK> and search for "Target
>>> > > > Version/s" = 3.4.1
>>> > > >
>>> > > > Committers should look at those and triage. Extremely important bug
>>> > > > fixes, documentation, and API tweaks that impact compatibility
>>> should
>>> > > > be worked on immediately. Everything else please retarget to an
>>> > > > appropriate release.
>>> > > >
>>> > > > ==
>>> > > > But my bug isn't fixed?
>>> > > > ==
>>> > > >
>>> > > > In order to make timely releases, we will typically not hold the
>>> > > > release unless the bug in question is a regression from the
>>> previous
>>> > > > release. That being said, if there is something which is a
>>> regression
>>> > > > that has not been correctly targeted please ping me or a committer
>>> to
>>> > > > help target the issue.
>>> > > >
>>> > >
>>> >
>>> > -
>>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> dev-unsubscr...@spark.apache.org>
>>> >
>>>
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> dev-unsubscr...@spark.apache.org>
>>>
>>>
>>>
>>>
>>>
>>>

Re: [VOTE] Release Spark 3.4.1 (RC1)

2023-06-21 Thread huaxin gao 
+1

On Tue, Jun 20, 2023 at 11:21 PM Hyukjin Kwon  wrote:

> +1
>
> On Wed, 21 Jun 2023 at 14:23, yangjie01  wrote:
>
>> +1
>>
>>
>> 在 2023/6/21 13:20，“L. C. Hsieh”> vii...@gmail.com>> 写入:
>>
>>
>> +1
>>
>>
>> On Tue, Jun 20, 2023 at 8:48 PM Dongjoon Hyun > > wrote:
>> >
>> > +1
>> >
>> > Dongjoon
>> >
>> > On 2023/06/20 02:51:32 Jia Fan wrote:
>> > > +1
>> > >
>> > > Dongjoon Hyun mailto:dongj...@apache.org>>
>> 于2023年6月20日周二 10:41写道：
>> > >
>> > > > Please vote on releasing the following candidate as Apache Spark
>> version
>> > > > 3.4.1.
>> > > >
>> > > > The vote is open until June 23rd 1AM (PST) and passes if a majority
>> +1 PMC
>> > > > votes are cast, with a minimum of 3 +1 votes.
>> > > >
>> > > > [ ] +1 Release this package as Apache Spark 3.4.1
>> > > > [ ] -1 Do not release this package because ...
>> > > >
>> > > > To learn more about Apache Spark, please see
>> https://spark.apache.org/ 
>> > > >
>> > > > The tag to be voted on is v3.4.1-rc1 (commit
>> > > > 6b1ff22dde1ead51cbf370be6e48a802daae58b6)
>> > > > https://github.com/apache/spark/tree/v3.4.1-rc1 <
>> https://github.com/apache/spark/tree/v3.4.1-rc1>
>> > > >
>> > > > The release files, including signatures, digests, etc. can be found
>> at:
>> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/ <
>> https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/>
>> > > >
>> > > > Signatures used for Spark RCs can be found in this file:
>> > > > https://dist.apache.org/repos/dist/dev/spark/KEYS <
>> https://dist.apache.org/repos/dist/dev/spark/KEYS>
>> > > >
>> > > > The staging repository for this release can be found at:
>> > > >
>> https://repository.apache.org/content/repositories/orgapachespark-1443/ <
>> https://repository.apache.org/content/repositories/orgapachespark-1443/>
>> > > >
>> > > > The documentation corresponding to this release can be found at:
>> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/ <
>> https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/>
>> > > >
>> > > > The list of bug fixes going into 3.4.1 can be found at the
>> following URL:
>> > > > https://issues.apache.org/jira/projects/SPARK/versions/12352874 <
>> https://issues.apache.org/jira/projects/SPARK/versions/12352874>
>> > > >
>> > > > This release is using the release script of the tag v3.4.1-rc1.
>> > > >
>> > > > FAQ
>> > > >
>> > > > =
>> > > > How can I help test this release?
>> > > > =
>> > > >
>> > > > If you are a Spark user, you can help us test this release by taking
>> > > > an existing Spark workload and running on this release candidate,
>> then
>> > > > reporting any regressions.
>> > > >
>> > > > If you're working in PySpark you can set up a virtual env and
>> install
>> > > > the current RC and see if anything important breaks, in the
>> Java/Scala
>> > > > you can add the staging repository to your projects resolvers and
>> test
>> > > > with the RC (make sure to clean up the artifact cache before/after
>> so
>> > > > you don't end up building with a out of date RC going forward).
>> > > >
>> > > > ===
>> > > > What should happen to JIRA tickets still targeting 3.4.1?
>> > > > ===
>> > > >
>> > > > The current list of open tickets targeted at 3.4.1 can be found at:
>> > > > https://issues.apache.org/jira/projects/SPARK <
>> https://issues.apache.org/jira/projects/SPARK> and search for "Target
>> > > > Version/s" = 3.4.1
>> > > >
>> > > > Committers should look at those and triage. Extremely important bug
>> > > > fixes, documentation, and API tweaks that impact compatibility
>> should
>> > > > be worked on immediately. Everything else please retarget to an
>> > > > appropriate release.
>> > > >
>> > > > ==
>> > > > But my bug isn't fixed?
>> > > > ==
>> > > >
>> > > > In order to make timely releases, we will typically not hold the
>> > > > release unless the bug in question is a regression from the previous
>> > > > release. That being said, if there is something which is a
>> regression
>> > > > that has not been correctly targeted please ping me or a committer
>> to
>> > > > help target the issue.
>> > > >
>> > >
>> >
>> > -
>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > dev-unsubscr...@spark.apache.org>
>> >
>>
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > dev-unsubscr...@spark.apache.org>
>>
>>
>>
>>
>>
>>

Re: [VOTE] Release Spark 3.4.1 (RC1)

2023-06-21 Thread Hyukjin Kwon 
+1

On Wed, 21 Jun 2023 at 14:23, yangjie01  wrote:

> +1
>
>
> 在 2023/6/21 13:20，“L. C. Hsieh”mailto:vii...@gmail.com>>
> 写入:
>
>
> +1
>
>
> On Tue, Jun 20, 2023 at 8:48 PM Dongjoon Hyun  > wrote:
> >
> > +1
> >
> > Dongjoon
> >
> > On 2023/06/20 02:51:32 Jia Fan wrote:
> > > +1
> > >
> > > Dongjoon Hyun mailto:dongj...@apache.org>>
> 于2023年6月20日周二 10:41写道：
> > >
> > > > Please vote on releasing the following candidate as Apache Spark
> version
> > > > 3.4.1.
> > > >
> > > > The vote is open until June 23rd 1AM (PST) and passes if a majority
> +1 PMC
> > > > votes are cast, with a minimum of 3 +1 votes.
> > > >
> > > > [ ] +1 Release this package as Apache Spark 3.4.1
> > > > [ ] -1 Do not release this package because ...
> > > >
> > > > To learn more about Apache Spark, please see
> https://spark.apache.org/ 
> > > >
> > > > The tag to be voted on is v3.4.1-rc1 (commit
> > > > 6b1ff22dde1ead51cbf370be6e48a802daae58b6)
> > > > https://github.com/apache/spark/tree/v3.4.1-rc1 <
> https://github.com/apache/spark/tree/v3.4.1-rc1>
> > > >
> > > > The release files, including signatures, digests, etc. can be found
> at:
> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/ <
> https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/>
> > > >
> > > > Signatures used for Spark RCs can be found in this file:
> > > > https://dist.apache.org/repos/dist/dev/spark/KEYS <
> https://dist.apache.org/repos/dist/dev/spark/KEYS>
> > > >
> > > > The staging repository for this release can be found at:
> > > >
> https://repository.apache.org/content/repositories/orgapachespark-1443/ <
> https://repository.apache.org/content/repositories/orgapachespark-1443/>
> > > >
> > > > The documentation corresponding to this release can be found at:
> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/ <
> https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/>
> > > >
> > > > The list of bug fixes going into 3.4.1 can be found at the following
> URL:
> > > > https://issues.apache.org/jira/projects/SPARK/versions/12352874 <
> https://issues.apache.org/jira/projects/SPARK/versions/12352874>
> > > >
> > > > This release is using the release script of the tag v3.4.1-rc1.
> > > >
> > > > FAQ
> > > >
> > > > =
> > > > How can I help test this release?
> > > > =
> > > >
> > > > If you are a Spark user, you can help us test this release by taking
> > > > an existing Spark workload and running on this release candidate,
> then
> > > > reporting any regressions.
> > > >
> > > > If you're working in PySpark you can set up a virtual env and install
> > > > the current RC and see if anything important breaks, in the
> Java/Scala
> > > > you can add the staging repository to your projects resolvers and
> test
> > > > with the RC (make sure to clean up the artifact cache before/after so
> > > > you don't end up building with a out of date RC going forward).
> > > >
> > > > ===
> > > > What should happen to JIRA tickets still targeting 3.4.1?
> > > > ===
> > > >
> > > > The current list of open tickets targeted at 3.4.1 can be found at:
> > > > https://issues.apache.org/jira/projects/SPARK <
> https://issues.apache.org/jira/projects/SPARK> and search for "Target
> > > > Version/s" = 3.4.1
> > > >
> > > > Committers should look at those and triage. Extremely important bug
> > > > fixes, documentation, and API tweaks that impact compatibility should
> > > > be worked on immediately. Everything else please retarget to an
> > > > appropriate release.
> > > >
> > > > ==
> > > > But my bug isn't fixed?
> > > > ==
> > > >
> > > > In order to make timely releases, we will typically not hold the
> > > > release unless the bug in question is a regression from the previous
> > > > release. That being said, if there is something which is a regression
> > > > that has not been correctly targeted please ping me or a committer to
> > > > help target the issue.
> > > >
> > >
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org  dev-unsubscr...@spark.apache.org>
> >
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org  dev-unsubscr...@spark.apache.org>
>
>
>
>
>
>