Re:Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

[beliefer]

+1




在 2024-04-15 15:54:07，"Peter Toth"  写道：

+1



Wenchen Fan  ezt írta (időpont: 2024. ápr. 15., H, 9:08):

+1



On Sun, Apr 14, 2024 at 6:28 AM Dongjoon Hyun  wrote:

I'll start from my +1.

Dongjoon.

On 2024/04/13 22:22:05 Dongjoon Hyun wrote:
> Please vote on SPARK-4 to use ANSI SQL mode by default.
> The technical scope is defined in the following PR which is
> one line of code change and one line of migration guide.
>
> - DISCUSSION:
> https://lists.apache.org/thread/ztlwoz1v1sn81ssks12tb19x37zozxlz
> - JIRA: https://issues.apache.org/jira/browse/SPARK-4
> - PR: https://github.com/apache/spark/pull/46013
>
> The vote is open until April 17th 1AM (PST) and passes
> if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Use ANSI SQL mode by default
> [ ] -1 Do not use ANSI SQL mode by default because ...
>
> Thank you in advance.
>
> Dongjoon
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re:[ANNOUNCE] Apache Spark 3.5.1 released

[beliefer]

Congratulations!







At 2024-02-28 17:43:25, "Jungtaek Lim"  wrote:

Hi everyone,


We are happy to announce the availability of Spark 3.5.1!

Spark 3.5.1 is a maintenance release containing stability fixes. This
release is based on the branch-3.5 maintenance branch of Spark. We strongly
recommend all 3.5 users to upgrade to this stable release.

To download Spark 3.5.1, head over to the download page:
https://spark.apache.org/downloads.html

To view the release notes:
https://spark.apache.org/releases/spark-release-3-5-1.html

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.

Jungtaek Lim



ps. Yikun is helping us through releasing the official docker image for Spark 
3.5.1 (Thanks Yikun!) It may take some time to be generally available.

Re:Re: [DISCUSS] Release Spark 3.5.1?

[beliefer]

+1







在 2024-02-04 15:26:13，"Dongjoon Hyun"  写道：

+1



On Sat, Feb 3, 2024 at 9:18 PM yangjie01  wrote:

+1

在 2024/2/4 13:13，“Kent Yao”mailto:y...@apache.org>> 写入:


+1


Jungtaek Lim mailto:kabhwan.opensou...@gmail.com>> 于2024年2月3日周六 21:14写道：
>
> Hi dev,
>
> looks like there are a huge number of commits being pushed to branch-3.5 
> after 3.5.0 was released, 200+ commits.
>
> $ git log --oneline v3.5.0..HEAD | wc -l
> 202
>
> Also, there are 180 JIRA tickets containing 3.5.1 as fixed version, and 10 
> resolved issues are either marked as blocker (even correctness issues) or 
> critical, which justifies the release.
> https://issues.apache.org/jira/projects/SPARK/versions/12353495 
> 
>
> What do you think about releasing 3.5.1 with the current head of branch-3.5? 
> I'm happy to volunteer as the release manager.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org 







-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re:[ANNOUNCE] Apache Spark 3.4.2 released

[beliefer]

Congratulations!







At 2023-12-01 01:23:55, "Dongjoon Hyun"  wrote:

We are happy to announce the availability of Apache Spark 3.4.2!

Spark 3.4.2 is a maintenance release containing many fixes including
security and correctness domains. This release is based on the
branch-3.4 maintenance branch of Spark. We strongly
recommend all 3.4 users to upgrade to this stable release.

To download Spark 3.4.2, head over to the download page:
https://spark.apache.org/downloads.html

To view the release notes:
https://spark.apache.org/releases/spark-release-3-4-2.html

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.

Dongjoon Hyun

Re:Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

[beliefer]

+1







At 2023-09-26 13:03:56, "Ruifeng Zheng"  wrote:

+1



On Tue, Sep 26, 2023 at 12:51 PM Hyukjin Kwon  wrote:

Hi all,

I would like to start the vote for updating documentation hosted for EOL and 
maintenance releases to improve the usability here, and in order for end users 
to read the proper and correct documentation.

For discussion thread, please refer to 
https://lists.apache.org/thread/1675rzxx5x4j2x03t9x0kfph8tlys0cx.


Here is one example:
- https://github.com/apache/spark/pull/42989
- https://github.com/apache/spark-website/pull/480


Starting with my own +1.

Re:Are DataFrame rows ordered without an explicit ordering clause?

[beliefer]

AFAIK, The order is free whether it's SQL without spcified ORDER BY clause or  
DataFrame without sort. The behavior is consistent between them.







At 2023-09-18 23:47:40, "Nicholas Chammas"  wrote:

I’ve always considered DataFrames to be logically equivalent to SQL tables or 
queries.


In SQL, the result order of any query is implementation-dependent without an 
explicit ORDER BY clause. Technically, you could run `SELECT * FROM table;` 10 
times in a row and get 10 different orderings.


I thought the same applied to DataFrames, but the docstring for the recently 
added method DataFrame.offset implies otherwise.


This example will work fine in practice, of course. But if DataFrames are 
technically unordered without an explicit ordering clause, then in theory a 
future implementation change may result in “Bob" being the “first” row in the 
DataFrame, rather than “Tom”. That would make the example incorrect.


Is that not the case?


Nick

Re:[ANNOUNCE] Apache Spark 3.5.0 released

[beliefer]

Congratulations! Apache Spark. 







At 2023-09-16 01:01:40, "Yuanjian Li"  wrote:

Hi All,


We are happy to announce the availability of Apache Spark 3.5.0!

Apache Spark 3.5.0 is the sixth release of the 3.x line.

To download Spark 3.5.0, head over to the download page:
https://spark.apache.org/downloads.html
(Please note: the PyPi upload is pending due to a size limit request; we're 
actively following up here with the PyPi organization)

To view the release notes:
https://spark.apache.org/releases/spark-release-3-5-0.html

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.

Best,
Yuanjian

Re:[ANNOUNCE] Apache Spark 3.4.1 released

[beliefer]

Thanks! Dongjoon Hyun.
Congratulation too!







At 2023-06-24 07:57:05, "Dongjoon Hyun"  wrote:

We are happy to announce the availability of Apache Spark 3.4.1!

Spark 3.4.1 is a maintenance release containing stability fixes. This
release is based on the branch-3.4 maintenance branch of Spark. We strongly
recommend all 3.4 users to upgrade to this stable release.

To download Spark 3.4.1, head over to the download page:
https://spark.apache.org/downloads.html

To view the release notes:
https://spark.apache.org/releases/spark-release-3-4-1.html

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.

Dongjoon Hyun

Re: Apache Spark 3.4.1 Release?

[beliefer]

Dongjoon. Thank you.
There is a issue should be fixed.
https://issues.apache.org/jira/browse/SPARK-44018







在 2023-06-12 13:22:30，"Dongjoon Hyun"  写道：

Thank you all.


I'll check and prepare `branch-3.4` for the target date, June 20th.


Dongjoon.




On Fri, Jun 9, 2023 at 10:47 PM yangjie01  wrote:


+1

 

Thank you Dongjoon ~

 

发件人: Ruifeng Zheng 
日期: 2023年6月10日星期六 09:39
收件人: Xiao Li 
抄送: Wenchen Fan , Xinrong Meng , dev 

主题: Re: Apache Spark 3.4.1 Release?

 

+1

 

Thank you Dongjoon!

 

 

On Fri, Jun 9, 2023 at 11:54 PM Xiao Li  wrote:

+1

 

On Fri, Jun 9, 2023 at 08:30 Wenchen Fan  wrote:

+1

 

On Fri, Jun 9, 2023 at 8:52 PM Xinrong Meng  wrote:

+1. Thank you Doonjoon!

 

Thanks,

 

Xinrong Meng

 

Mridul Muralidharan 于2023年6月9日 周五上午5:22写道：

 

+1, thanks Dongjoon !

 

Regards,

Mridul 

 

On Thu, Jun 8, 2023 at 7:16 PM Jia Fan  wrote:

+1






Jia Fan








2023年6月9日 08:00，Yuming Wang  写道：

 

+1.

 

On Fri, Jun 9, 2023 at 7:14 AM Chao Sun  wrote:

+1 too

On Thu, Jun 8, 2023 at 2:34 PM kazuyuki tanimura
 wrote:
>
> +1 (non-binding), Thank you Dongjoon
>
> Kazu
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

 

--

Re:[VOTE] Release Apache Spark 3.4.0 (RC7)

[beliefer]

+1




At 2023-04-08 07:29:46, "Xinrong Meng"  wrote:

Please vote on releasing the following candidate(RC7) as Apache Spark version 
3.4.0.

The vote is open until 11:59pm Pacific time April 12th and passes if a majority 
+1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.4.0-rc7 (commit 
87a5442f7ed96b11051d8a9333476d080054e5a0):
https://github.com/apache/spark/tree/v3.4.0-rc7

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc7-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1441

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc7-docs/

The list of bug fixes going into 3.4.0 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12351465

This release is using the release script of the tag v3.4.0-rc7.


FAQ

=
How can I help test this release?
=
If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===
What should happen to JIRA tickets still targeting 3.4.0?
===
The current list of open tickets targeted at 3.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" 
= 3.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==
But my bug isn't fixed?
==
In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Thanks,
Xinrong Meng

Re:Re: [VOTE] Release Apache Spark 3.4.0 (RC4)

[beliefer]

There is a bug fix.

https://issues.apache.org/jira/browse/SPARK-42740







在 2023-03-10 20:48:30，"Xinrong Meng"  写道：

https://issues.apache.org/jira/browse/SPARK-42745 can be a new release blocker, 
thanks @Peter Toth for reporting that.


On Fri, Mar 10, 2023 at 8:21 PM Xinrong Meng  wrote:

Please vote on releasing the following candidate(RC4) as Apache Spark version 
3.4.0.

The vote is open until 11:59pm Pacific time March 15th and passes if a majority 
+1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.4.0-rc4 (commit 
4000d6884ce973eb420e871c8d333431490be763):
https://github.com/apache/spark/tree/v3.4.0-rc4

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc4-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1438

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc4-docs/

The list of bug fixes going into 3.4.0 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12351465

This release is using the release script of the tag v3.4.0-rc4.


FAQ

=
How can I help test this release?
=
If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===
What should happen to JIRA tickets still targeting 3.4.0?
===
The current list of open tickets targeted at 3.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" 
= 3.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==
But my bug isn't fixed?
==
In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Thanks,
Xinrong Meng

Re:[ANNOUNCE] Apache Spark 3.3.2 released

[beliefer]

Congratulations !





At 2023-02-17 16:58:22, "L. C. Hsieh"  wrote:
>We are happy to announce the availability of Apache Spark 3.3.2!
>
>Spark 3.3.2 is a maintenance release containing stability fixes. This
>release is based on the branch-3.3 maintenance branch of Spark. We strongly
>recommend all 3.3 users to upgrade to this stable release.
>
>To download Spark 3.3.2, head over to the download page:
>https://spark.apache.org/downloads.html
>
>To view the release notes:
>https://spark.apache.org/releases/spark-release-3-3-2.html
>
>We would like to acknowledge all community members for contributing to this
>release. This release would not have been possible without you.
>
>-
>To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re:[ANNOUNCE] Apache Spark 3.3.1 released

[beliefer]

Congratulations everyone have contributed to this release.




At 2022-10-26 14:21:36, "Yuming Wang"  wrote:

We are happy to announce the availability of Apache Spark 3.3.1!

Spark 3.3.1 is a maintenance release containing stability fixes. This
release is based on the branch-3.3 maintenance branch of Spark. We strongly
recommend all 3.3 users to upgrade to this stable release.

To download Spark 3.3.1, head over to the download page:
https://spark.apache.org/downloads.html

To view the release notes:
https://spark.apache.org/releases/spark-release-3-3-1.html

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.

Re:[DISCUSS] Improve the performance of Spark Decimal.

[beliefer]

There is a WIP PR https://github.com/apache/spark/pull/37536, only implement 
the add operator of Decimal128







At 2022-08-24 12:09:03, "beliefer"  wrote:

Hi all,




Recently, we found many SQL query could improve performance by replace Spark 
Decimal to Double. This is confirmed by our many investigations and tests. We 
have investigated other databases or big data systems and found that some 
projects also have problems or PR for improving the performance of Java 
BigDecimal.




Please refer:




Hive: 
https://github.com/apache/hive/commit/98893823dc57a9c187ad5c97c9c8ce03af1fa734#diff-b4a1ba785de48d9b18f680e38c39c76e86d939c7c7a2dc75be776ff6b11999df




Hive uses an int array with a length of 4 to implement the new Decimal128 type.




Trino: https://github.com/trinodb/trino/pull/10051




Trino references the github project https://github.com/martint/int128. The 
project int128 uses an long array with a length of 2 to implement the Int128 
type. Trino further encapsulates Int128 and implements the new decimal type.




In combination with the memory format of spark SQL, Hive not only stores 4 ints 
to unsaferow, but also stores one byte for positive and negative, and one short 
for scale.The Trino method only needs to store 2 long to unsaferow in 
succession, and needs to store an int to represent scale. Because UnsafeRow 
store one 8-byte word per field, so the Hive method need store 6 * 8 = 48 
bytes, but Trino method only need stroe 3 * 8 = 24 bytes.




Considering that we can express the positive and negative and scale information 
required by Hive method through type metadata, we still need to occupy 4 * 8 = 
32 bytes. At the same time, we can store the scale information required by the 
Trino method into the type metadata, so we only need to occupy 2 * 8 = 16 bytes.




We already implement a new Decimal128 type with Int128 and implement the + 
operator of Decimal128 in my forked Spark. After the performance comparison 
between Decimal128(Int128) and Spark Decimal, we get the benchmark and know 
Decimal128 has 1.5x ~ 3x performance improvement compared with Spark Decimal.




The attachment of the email contains relevant design documents and benchmarks.




What do you think of this idea? 

If you support this idea, I could create a first PR for further and deeper 
discussion.

Re:Re: [VOTE][SPIP] Spark Connect

[beliefer]

+1
Yeah, I tried to use Apache Livy, so as we can runing interactive query. But 
the Spark Driver in Livy looks heavy.

The SPIP may resolve the issue.







At 2022-06-14 18:11:21, "Wenchen Fan"  wrote:

+1



On Tue, Jun 14, 2022 at 9:38 AM Ruifeng Zheng  wrote:

+1





-- 原始邮件 --
发件人: "huaxin gao" ;
发送时间: 2022年6月14日(星期二) 上午8:47
收件人: "L. C. Hsieh";
抄送: "Spark dev list";
主题: Re: [VOTE][SPIP] Spark Connect


+1



On Mon, Jun 13, 2022 at 5:42 PM L. C. Hsieh  wrote:

+1

On Mon, Jun 13, 2022 at 5:41 PM Chao Sun  wrote:
>
> +1 (non-binding)
>
> On Mon, Jun 13, 2022 at 5:11 PM Hyukjin Kwon  wrote:
>>
>> +1
>>
>> On Tue, 14 Jun 2022 at 08:50, Yuming Wang  wrote:
>>>
>>> +1.
>>>
>>> On Tue, Jun 14, 2022 at 2:20 AM Matei Zaharia  
>>> wrote:

 +1, very excited about this direction.

 Matei

 On Jun 13, 2022, at 11:07 AM, Herman van Hovell 
  wrote:

 Let me kick off the voting...

 +1

 On Mon, Jun 13, 2022 at 2:02 PM Herman van Hovell  
 wrote:
>
> Hi all,
>
> I’d like to start a vote for SPIP: "Spark Connect"
>
> The goal of the SPIP is to introduce a Dataframe based client/server API 
> for Spark
>
> Please also refer to:
>
> - Previous discussion in dev mailing list: [DISCUSS] SPIP: Spark Connect 
> - A client and server interface for Apache Spark.
> - Design doc: Spark Connect - A client and server interface for Apache 
> Spark.
> - JIRA: SPARK-39375
>
> Please vote on the SPIP for the next 72 hours:
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don’t think this is a good idea because …
>
> Kind Regards,
> Herman



-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re:回复： [VOTE] Release Spark 3.3.0 (RC6)

[beliefer]

+1 AFAIK, no blocking issues now.
Glad to hear to release 3.3.0 !




在 2022-06-14 09:38:35，"Ruifeng Zheng"  写道：

+1 (non-binding)


Maxim, thank you for driving this release!


thanks,
ruifeng







-- 原始邮件 --
发件人: "Chao Sun" ;
发送时间: 2022年6月14日(星期二) 上午8:45
收件人: "Cheng Su";
抄送: "L. C. Hsieh";"dev";
主题: Re: [VOTE] Release Spark 3.3.0 (RC6)


+1 (non-binding)


Thanks,
Chao


On Mon, Jun 13, 2022 at 5:37 PM Cheng Su  wrote:


+1 (non-binding).

 

Thanks,

Cheng Su

 

From: L. C. Hsieh 
Date: Monday, June 13, 2022 at 5:13 PM
To: dev 
Subject: Re: [VOTE] Release Spark 3.3.0 (RC6)

+1

On Mon, Jun 13, 2022 at 5:07 PM Holden Karau  wrote:
>
> +1
>
> On Mon, Jun 13, 2022 at 4:51 PM Yuming Wang  wrote:
>>
>> +1 (non-binding)
>>
>> On Tue, Jun 14, 2022 at 7:41 AM Dongjoon Hyun  
>> wrote:
>>>
>>> +1
>>>
>>> Thanks,
>>> Dongjoon.
>>>
>>> On Mon, Jun 13, 2022 at 3:54 PM Chris Nauroth  wrote:

 +1 (non-binding)

 I repeated all checks I described for RC5:

 https://lists.apache.org/thread/ksoxmozgz7q728mnxl6c2z7ncmo87vls

 Maxim, thank you for your dedication on these release candidates.

 Chris Nauroth


 On Mon, Jun 13, 2022 at 3:21 PM Mridul Muralidharan  
 wrote:
>
>
> +1
>
> Signatures, digests, etc check out fine.
> Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes
>
> The test "SPARK-33084: Add jar support Ivy URI in SQL" in 
> sql.SQLQuerySuite fails; but other than that, rest looks good.
>
> Regards,
> Mridul
>
>
>
> On Mon, Jun 13, 2022 at 4:25 PM Tom Graves  
> wrote:
>>
>> +1
>>
>> Tom
>>
>> On Thursday, June 9, 2022, 11:27:50 PM CDT, Maxim Gekk 
>>  wrote:
>>
>>
>> Please vote on releasing the following candidate as Apache Spark version 
>> 3.3.0.
>>
>> The vote is open until 11:59pm Pacific time June 14th and passes if a 
>> majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 3.3.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v3.3.0-rc6 (commit 
>> f74867bddfbcdd4d08076db36851e88b15e66556):
>> https://github.com/apache/spark/tree/v3.3.0-rc6
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc6-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1407
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc6-docs/
>>
>> The list of bug fixes going into 3.3.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12350369
>>
>> This release is using the release script of the tag v3.3.0-rc6.
>>
>>
>> FAQ
>>
>> =
>> How can I help test this release?
>> =
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===
>> What should happen to JIRA tickets still targeting 3.3.0?
>> ===
>> The current list of open tickets targeted at 3.3.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK  and search for "Target 
>> Version/s" = 3.3.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==
>> But my bug isn't fixed?
>> ==
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>> 

Re:Re: Unable to create view due to up cast error when migrating from Hive to Spark

[beliefer]

Thank you for the reply !




At 2022-05-18 20:27:27, "Wenchen Fan"  wrote:

A view is essentially a SQL query. It's fragile to share views between Spark 
and Hive because different systems have different SQL dialects. They may 
interpret the view SQL query differently and introduce unexpected behaviors.


In this case, Spark returns decimal type for gender * 0.3 - 0.1 but Hive 
returns double type. The view schema was determined during creation by Hive, 
which does not match the view SQL query when we use Spark to read the view. We 
need to re-create this view using Spark. Actually I think we need to do the 
same for every Hive view if we need to use it in Spark.


On Wed, May 18, 2022 at 7:03 PM beliefer  wrote:


During the migration from hive to spark, there was a problem with the SQL used 
to create views in hive. The problem is that the SQL that legally creates a 
view in hive will make an error when executed in spark SQL.

The SQL is as follows:

CREATE VIEW test_db.my_view AS
select
case
when age > 12 then gender * 0.3 - 0.1
end AS TT,
gender,
age,
careers,
education
from
test_db.my_table;

The error message is as follows:

Cannot up cast TT from decimal(13, 1) to double.

The type path of the target object is:



You can either add an explicit cast to the input data or choose a higher 
precision type of the field in the target object



How should we solve this problem?




 

Unable to create view due to up cast error when migrating from Hive to Spark

[beliefer]

During the migration from hive to spark, there was a problem with the SQL used 
to create views in hive. The problem is that the SQL that legally creates a 
view in hive will make an error when executed in spark SQL.

The SQL is as follows:

CREATE VIEW test_db.my_view AS
select
case
when age > 12 then gender * 0.3 - 0.1
end AS TT,
gender,
age,
careers,
education
from
test_db.my_table;

The error message is as follows:

Cannot up cast TT from decimal(13, 1) to double.

The type path of the target object is:



You can either add an explicit cast to the input data or choose a higher 
precision type of the field in the target object



How should we solve this problem?

Unable to create view due to up cast error when migrating from Hive to Spark

[beliefer]

During the migration from Hive to spark, there was a problem when the view 
created in Hive was used in Spark SQL.
The origin Hive SQL show below:


CREATE VIEW myView AS

SELECT
CASE WHEN age > 12 THEN CAST(gender * 0.3 - 0.1 AS double) END AS TT, gender, 
age

FROM myTable;

Users use Spark SQL to query the view, but encountered up cast error. The error 
message is as follows:

Cannot up cast TT from decimal(13, 1) to double.

The type path of the target object is:



You can either add an explicit cast to the input data or choose a higher 
precision type of the field in the target object

How should we solve this problem?

Re:Re: Re: Re: [VOTE] Release Spark 3.3.0 (RC2)

[beliefer]

OK. let it into 3.3.1




在 2022-05-17 18:59:13，"Hyukjin Kwon"  写道：

I think most users won't be affected since aggregate pushdown is disabled by 
default.


On Tue, 17 May 2022 at 19:53, beliefer  wrote:


If we not contains https://github.com/apache/spark/pull/36556, we will break 
change when we merge it into 3.3.1

At 2022-05-17 18:26:12, "Hyukjin Kwon"  wrote:

We need add https://github.com/apache/spark/pull/36556 to RC2.

We will likely have to change the version being added if RC2 passes.
Since this is a new API/improvement, I would prefer to not block the release by 
that.



On Tue, 17 May 2022 at 19:19, beliefer  wrote:

We need add https://github.com/apache/spark/pull/36556 to RC2.




在 2022-05-17 17:37:13，"Hyukjin Kwon"  写道：

That seems like a test-only issue. I made a quick followup at 
https://github.com/apache/spark/pull/36576.


On Tue, 17 May 2022 at 03:56, Sean Owen  wrote:

I'm still seeing failures related to the function registry, like:


ExpressionsSchemaSuite:
- Check schemas for expression examples *** FAILED ***
  396 did not equal 398 Expected 396 blocks in result file but got 398. Try 
regenerating the result files. (ExpressionsSchemaSuite.scala:161)

- SPARK-14415: All functions should have own descriptions *** FAILED ***
  "Function: bloom_filter_aggClass: 
org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregateUsage: 
N/A." contained "N/A." Failed for [function_desc: string] (N/A. existed in the 
result) (QueryTest.scala:54)



There seems to be consistently a difference of 2 in the list of expected 
functions and actual. I haven't looked closely, don't know this code. I'm on 
Ubuntu 22.04. Anyone else seeing something like this? Wondering if it's 
something weird to do with case sensitivity, hidden files lurking somewhere, 
etc.


I suspect it's not a 'real' error as the Linux-based testers work fine, but I 
also can't think of why this is failing.






On Mon, May 16, 2022 at 7:44 AM Maxim Gekk  
wrote:

Please vote on releasing the following candidate as Apache Spark version 3.3.0.



The vote is open until 11:59pm Pacific time May 19th and passes if a majority 
+1 PMC votes are cast, with a minimum of 3 +1 votes.



[ ] +1 Release this package as Apache Spark 3.3.0

[ ] -1 Do not release this package because ...



To learn more about Apache Spark, please see http://spark.apache.org/



The tag to be voted on is v3.3.0-rc2 (commit 
c8c657b922ac8fd8dcf9553113e11a80079db059):

https://github.com/apache/spark/tree/v3.3.0-rc2



The release files, including signatures, digests, etc. can be found at:

https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc2-bin/



Signatures used for Spark RCs can be found in this file:

https://dist.apache.org/repos/dist/dev/spark/KEYS



The staging repository for this release can be found at:

https://repository.apache.org/content/repositories/orgapachespark-1403



The documentation corresponding to this release can be found at:

https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc2-docs/



The list of bug fixes going into 3.3.0 can be found at the following URL:

https://issues.apache.org/jira/projects/SPARK/versions/12350369


This release is using the release script of the tag v3.3.0-rc2.





FAQ



=

How can I help test this release?

=

If you are a Spark user, you can help us test this release by taking

an existing Spark workload and running on this release candidate, then

reporting any regressions.



If you're working in PySpark you can set up a virtual env and install

the current RC and see if anything important breaks, in the Java/Scala

you can add the staging repository to your projects resolvers and test

with the RC (make sure to clean up the artifact cache before/after so

you don't end up building with a out of date RC going forward).



===

What should happen to JIRA tickets still targeting 3.3.0?

===

The current list of open tickets targeted at 3.3.0 can be found at:

https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" 
= 3.3.0



Committers should look at those and triage. Extremely important bug

fixes, documentation, and API tweaks that impact compatibility should

be worked on immediately. Everything else please retarget to an

appropriate release.



==

But my bug isn't fixed?

==

In order to make timely releases, we will typically not hold the

release unless the bug in question is a regression from the previous

release. That being said, if there is something which is a regression

that has not been correctly targeted please ping me or a committer to

help target the issue.


Maxim Gekk


Software Engineer

Databricks, Inc.





 





 

Re:Re: Re: [VOTE] Release Spark 3.3.0 (RC2)

[beliefer]

If we not contains https://github.com/apache/spark/pull/36556, we will break 
change when we merge it into 3.3.1

At 2022-05-17 18:26:12, "Hyukjin Kwon"  wrote:

We need add https://github.com/apache/spark/pull/36556 to RC2.

We will likely have to change the version being added if RC2 passes.
Since this is a new API/improvement, I would prefer to not block the release by 
that.



On Tue, 17 May 2022 at 19:19, beliefer  wrote:

We need add https://github.com/apache/spark/pull/36556 to RC2.




在 2022-05-17 17:37:13，"Hyukjin Kwon"  写道：

That seems like a test-only issue. I made a quick followup at 
https://github.com/apache/spark/pull/36576.


On Tue, 17 May 2022 at 03:56, Sean Owen  wrote:

I'm still seeing failures related to the function registry, like:


ExpressionsSchemaSuite:
- Check schemas for expression examples *** FAILED ***
  396 did not equal 398 Expected 396 blocks in result file but got 398. Try 
regenerating the result files. (ExpressionsSchemaSuite.scala:161)

- SPARK-14415: All functions should have own descriptions *** FAILED ***
  "Function: bloom_filter_aggClass: 
org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregateUsage: 
N/A." contained "N/A." Failed for [function_desc: string] (N/A. existed in the 
result) (QueryTest.scala:54)



There seems to be consistently a difference of 2 in the list of expected 
functions and actual. I haven't looked closely, don't know this code. I'm on 
Ubuntu 22.04. Anyone else seeing something like this? Wondering if it's 
something weird to do with case sensitivity, hidden files lurking somewhere, 
etc.


I suspect it's not a 'real' error as the Linux-based testers work fine, but I 
also can't think of why this is failing.






On Mon, May 16, 2022 at 7:44 AM Maxim Gekk  
wrote:

Please vote on releasing the following candidate as Apache Spark version 3.3.0.



The vote is open until 11:59pm Pacific time May 19th and passes if a majority 
+1 PMC votes are cast, with a minimum of 3 +1 votes.



[ ] +1 Release this package as Apache Spark 3.3.0

[ ] -1 Do not release this package because ...



To learn more about Apache Spark, please see http://spark.apache.org/



The tag to be voted on is v3.3.0-rc2 (commit 
c8c657b922ac8fd8dcf9553113e11a80079db059):

https://github.com/apache/spark/tree/v3.3.0-rc2



The release files, including signatures, digests, etc. can be found at:

https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc2-bin/



Signatures used for Spark RCs can be found in this file:

https://dist.apache.org/repos/dist/dev/spark/KEYS



The staging repository for this release can be found at:

https://repository.apache.org/content/repositories/orgapachespark-1403



The documentation corresponding to this release can be found at:

https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc2-docs/



The list of bug fixes going into 3.3.0 can be found at the following URL:

https://issues.apache.org/jira/projects/SPARK/versions/12350369


This release is using the release script of the tag v3.3.0-rc2.





FAQ



=

How can I help test this release?

=

If you are a Spark user, you can help us test this release by taking

an existing Spark workload and running on this release candidate, then

reporting any regressions.



If you're working in PySpark you can set up a virtual env and install

the current RC and see if anything important breaks, in the Java/Scala

you can add the staging repository to your projects resolvers and test

with the RC (make sure to clean up the artifact cache before/after so

you don't end up building with a out of date RC going forward).



===

What should happen to JIRA tickets still targeting 3.3.0?

===

The current list of open tickets targeted at 3.3.0 can be found at:

https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" 
= 3.3.0



Committers should look at those and triage. Extremely important bug

fixes, documentation, and API tweaks that impact compatibility should

be worked on immediately. Everything else please retarget to an

appropriate release.



==

But my bug isn't fixed?

==

In order to make timely releases, we will typically not hold the

release unless the bug in question is a regression from the previous

release. That being said, if there is something which is a regression

that has not been correctly targeted please ping me or a committer to

help target the issue.


Maxim Gekk


Software Engineer

Databricks, Inc.





 

Re:Re: [VOTE] Release Spark 3.3.0 (RC2)

[beliefer]

We need add https://github.com/apache/spark/pull/36556 to RC2.




在 2022-05-17 17:37:13，"Hyukjin Kwon"  写道：

That seems like a test-only issue. I made a quick followup at 
https://github.com/apache/spark/pull/36576.


On Tue, 17 May 2022 at 03:56, Sean Owen  wrote:

I'm still seeing failures related to the function registry, like:


ExpressionsSchemaSuite:
- Check schemas for expression examples *** FAILED ***
  396 did not equal 398 Expected 396 blocks in result file but got 398. Try 
regenerating the result files. (ExpressionsSchemaSuite.scala:161)

- SPARK-14415: All functions should have own descriptions *** FAILED ***
  "Function: bloom_filter_aggClass: 
org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregateUsage: 
N/A." contained "N/A." Failed for [function_desc: string] (N/A. existed in the 
result) (QueryTest.scala:54)



There seems to be consistently a difference of 2 in the list of expected 
functions and actual. I haven't looked closely, don't know this code. I'm on 
Ubuntu 22.04. Anyone else seeing something like this? Wondering if it's 
something weird to do with case sensitivity, hidden files lurking somewhere, 
etc.


I suspect it's not a 'real' error as the Linux-based testers work fine, but I 
also can't think of why this is failing.






On Mon, May 16, 2022 at 7:44 AM Maxim Gekk  
wrote:

Please vote on releasing the following candidate as Apache Spark version 3.3.0.



The vote is open until 11:59pm Pacific time May 19th and passes if a majority 
+1 PMC votes are cast, with a minimum of 3 +1 votes.



[ ] +1 Release this package as Apache Spark 3.3.0

[ ] -1 Do not release this package because ...



To learn more about Apache Spark, please see http://spark.apache.org/



The tag to be voted on is v3.3.0-rc2 (commit 
c8c657b922ac8fd8dcf9553113e11a80079db059):

https://github.com/apache/spark/tree/v3.3.0-rc2



The release files, including signatures, digests, etc. can be found at:

https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc2-bin/



Signatures used for Spark RCs can be found in this file:

https://dist.apache.org/repos/dist/dev/spark/KEYS



The staging repository for this release can be found at:

https://repository.apache.org/content/repositories/orgapachespark-1403



The documentation corresponding to this release can be found at:

https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc2-docs/



The list of bug fixes going into 3.3.0 can be found at the following URL:

https://issues.apache.org/jira/projects/SPARK/versions/12350369


This release is using the release script of the tag v3.3.0-rc2.





FAQ



=

How can I help test this release?

=

If you are a Spark user, you can help us test this release by taking

an existing Spark workload and running on this release candidate, then

reporting any regressions.



If you're working in PySpark you can set up a virtual env and install

the current RC and see if anything important breaks, in the Java/Scala

you can add the staging repository to your projects resolvers and test

with the RC (make sure to clean up the artifact cache before/after so

you don't end up building with a out of date RC going forward).



===

What should happen to JIRA tickets still targeting 3.3.0?

===

The current list of open tickets targeted at 3.3.0 can be found at:

https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" 
= 3.3.0



Committers should look at those and triage. Extremely important bug

fixes, documentation, and API tweaks that impact compatibility should

be worked on immediately. Everything else please retarget to an

appropriate release.



==

But my bug isn't fixed?

==

In order to make timely releases, we will typically not hold the

release unless the bug in question is a regression from the previous

release. That being said, if there is something which is a regression

that has not been correctly targeted please ping me or a committer to

help target the issue.


Maxim Gekk


Software Engineer

Databricks, Inc.

Unable to create view due to up cast error when migrating from Hive to Spark

[beliefer]

During the migration from hive to spark, there was a problem with the SQL used 
to create views in hive. The problem is that the SQL that legally creates a 
view in hive will make an error when executed in spark SQL.

The SQL is as follows:

CREATE VIEW myView AS

SELECT
CASE WHEN age > 12 THEN CAST(gender * 0.3 - 0.1 AS double) END AS TT, gender, 
age

FROM myTable;

The error message is as follows:

Cannot up cast TT from decimal(13, 1) to double.

The type path of the target object is:



You can either add an explicit cast to the input data or choose a higher 
precision type of the field in the target object



How should we solve this problem?

Re:[VOTE] Release Spark 3.3.0 (RC1)

[beliefer]

 @Maxim Gekk  Glad to hear that!


But there is a bug https://github.com/apache/spark/pull/36457
I think we should merge it into 3.3.0




At 2022-05-05 19:00:27, "Maxim Gekk"  wrote:

Please vote on releasing the following candidate as Apache Spark version 3.3.0.



The vote is open until 11:59pm Pacific time May 10th and passes if a majority 
+1 PMC votes are cast, with a minimum of 3 +1 votes.



[ ] +1 Release this package as Apache Spark 3.3.0

[ ] -1 Do not release this package because ...



To learn more about Apache Spark, please see http://spark.apache.org/



The tag to be voted on is v3.3.0-rc1 (commit 
482b7d54b522c4d1e25f3e84eabbc78126f22a3d):

https://github.com/apache/spark/tree/v3.3.0-rc1



The release files, including signatures, digests, etc. can be found at:

https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc1-bin/



Signatures used for Spark RCs can be found in this file:

https://dist.apache.org/repos/dist/dev/spark/KEYS



The staging repository for this release can be found at:

https://repository.apache.org/content/repositories/orgapachespark-1402



The documentation corresponding to this release can be found at:

https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc1-docs/



The list of bug fixes going into 3.3.0 can be found at the following URL:

https://issues.apache.org/jira/projects/SPARK/versions/12350369


This release is using the release script of the tag v3.3.0-rc1.





FAQ



=

How can I help test this release?

=

If you are a Spark user, you can help us test this release by taking

an existing Spark workload and running on this release candidate, then

reporting any regressions.



If you're working in PySpark you can set up a virtual env and install

the current RC and see if anything important breaks, in the Java/Scala

you can add the staging repository to your projects resolvers and test

with the RC (make sure to clean up the artifact cache before/after so

you don't end up building with a out of date RC going forward).



===

What should happen to JIRA tickets still targeting 3.3.0?

===

The current list of open tickets targeted at 3.3.0 can be found at:

https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" 
= 3.3.0



Committers should look at those and triage. Extremely important bug

fixes, documentation, and API tweaks that impact compatibility should

be worked on immediately. Everything else please retarget to an

appropriate release.



==

But my bug isn't fixed?

==

In order to make timely releases, we will typically not hold the

release unless the bug in question is a regression from the previous

release. That being said, if there is something which is a regression

that has not been correctly targeted please ping me or a committer to

help target the issue.


Maxim Gekk


Software Engineer

Databricks, Inc.

Re:Apache Spark 3.3 Release

[beliefer]

+1 Glad to see we will release 3.3.0.




At 2022-03-04 02:44:37, "Maxim Gekk"  wrote:

Hello All,

I would like to bring on the table the theme about the new Spark release 3.3. 
According to the public schedule at 
https://spark.apache.org/versioning-policy.html, we planned to start the code 
freeze and release branch cut on March 15th, 2022. Since this date is coming 
soon, I would like to take your attention on the topic and gather objections 
that you might have.

Bellow is the list of ongoing and active SPIPs:

Spark SQL:
- [SPARK-31357] DataSourceV2: Catalog API for view metadata
- [SPARK-35801] Row-level operations in Data Source V2
- [SPARK-37166] Storage Partitioned Join

Spark Core:
- [SPARK-20624] Add better handling for node shutdown
- [SPARK-25299] Use remote storage for persisting shuffle data

PySpark:
- [SPARK-26413] RDD Arrow Support in Spark Core and PySpark

Kubernetes:
- [SPARK-36057] Support Customized Kubernetes Schedulers

Probably, we should finish if there are any remaining works for Spark 3.3, and 
switch to QA mode, cut a branch and keep everything on track. I would like to 
volunteer to help drive this process.



Best regards,
Max Gekk

Re:[ANNOUNCE] Apache Spark 3.2.1 released

[beliefer]

Thank you huaxin gao!
Glad to see the release.







At 2022-01-29 09:07:13, "huaxin gao"  wrote:

We are happy to announce the availability of Spark 3.2.1!

Spark 3.2.1 is a maintenance release containing stability fixes. This
release is based on the branch-3.2 maintenance branch of Spark. We strongly
recommend all 3.2 users to upgrade to this stable release.

To download Spark 3.2.1, head over to the download page:
https://spark.apache.org/downloads.html

To view the release notes:
https://spark.apache.org/releases/spark-release-3-2-1.html

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.



Huaxin Gao

Re:[VOTE] Release Spark 3.2.1 (RC1)

[beliefer]

+1







At 2022-01-11 02:09:46, "huaxin gao"  wrote:

Please vote on releasing the following candidate as Apache Spark version 3.2.1.

The vote is open until Jan. 13th at 12 PM PST (8 PM UTC) and passes if a 
majority 
+1 PMC votes are cast, with a minimum of 3 + 1 votes.

[ ] +1 Release this package as Apache Spark 3.2.1
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

There are currently no issues targeting 3.2.1 (try project = SPARK AND
"Target Version/s" = "3.2.1" AND status in (Open, Reopened, "In Progress"))

The tag to be voted on is v3.2.1-rc1 (commit
2b0ee226f8dd17b278ad11139e62464433191653):
https://github.com/apache/spark/tree/v3.2.1-rc1

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc1-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1395/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc1-docs/

The list of bug fixes going into 3.2.1 can be found at the following URL:
https://s.apache.org/7tzik

This release is using the release script of the tag v3.2.1-rc1.

FAQ


=
How can I help test this release?
=

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===
What should happen to JIRA tickets still targeting 3.2.1?
===

The current list of open tickets targeted at 3.2.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target
Version/s" = 3.2.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==
But my bug isn't fixed?
==

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Re: Spark-3.0 - performance degradation

[beliefer]

I test it and cannot reproduce the issue.
I build Spark-3.1.0 and Spark2.3.1.
After many tests, it is found that there is little difference between them,
and they win and lose each other.
And from the view of event timeline, Spark-3.1.0 looks more accurate.



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Spark-3.0 - performance degradation

[beliefer]

Can you provide configuration information?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Spark-3.0 - performance degradation

[beliefer]

Can you provide configuration information?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Spark-3.0 - performance degradation

[beliefer]

Can you show the running configuration information?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org