Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

[Xiao Li]

+1

Xiao

Felix Cheung  于2019年8月30日周五 上午2:03写道：

> +1
>
> Run tests, R tests, r-hub Debian, Ubuntu, mac, Windows
>
> --
> *From:* Hyukjin Kwon 
> *Sent:* Wednesday, August 28, 2019 9:14 PM
> *To:* Takeshi Yamamuro
> *Cc:* dev; Dongjoon Hyun
> *Subject:* Re: [VOTE] Release Apache Spark 2.4.4 (RC3)
>
> +1 (from the last blocker PR)
>
> 2019년 8월 29일 (목) 오전 8:20, Takeshi Yamamuro 님이 작성:
>
>> I checked the tests passed again on the same env.
>> It looks ok.
>>
>>
>> On Thu, Aug 29, 2019 at 6:15 AM Marcelo Vanzin
>>  wrote:
>>
>>> +1
>>>
>>> On Tue, Aug 27, 2019 at 4:06 PM Dongjoon Hyun 
>>> wrote:
>>> >
>>> > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.4.
>>> >
>>> > The vote is open until August 30th 5PM PST and passes if a majority +1
>>> PMC votes are cast, with a minimum of 3 +1 votes.
>>> >
>>> > [ ] +1 Release this package as Apache Spark 2.4.4
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > To learn more about Apache Spark, please see http://spark.apache.org/
>>> >
>>> > The tag to be voted on is v2.4.4-rc3 (commit
>>> 7955b3962ac46b89564e0613db7bea98a1478bf2):
>>> > https://github.com/apache/spark/tree/v2.4.4-rc3
>>> >
>>> > The release files, including signatures, digests, etc. can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/
>>> >
>>> > Signatures used for Spark RCs can be found in this file:
>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> >
>>> > The staging repository for this release can be found at:
>>> >
>>> https://repository.apache.org/content/repositories/orgapachespark-1332/
>>> >
>>> > The documentation corresponding to this release can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/
>>> >
>>> > The list of bug fixes going into 2.4.4 can be found at the following
>>> URL:
>>> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
>>> >
>>> > This release is using the release script of the tag v2.4.4-rc3.
>>> >
>>> > FAQ
>>> >
>>> > =
>>> > How can I help test this release?
>>> > =
>>> >
>>> > If you are a Spark user, you can help us test this release by taking
>>> > an existing Spark workload and running on this release candidate, then
>>> > reporting any regressions.
>>> >
>>> > If you're working in PySpark you can set up a virtual env and install
>>> > the current RC and see if anything important breaks, in the Java/Scala
>>> > you can add the staging repository to your projects resolvers and test
>>> > with the RC (make sure to clean up the artifact cache before/after so
>>> > you don't end up building with a out of date RC going forward).
>>> >
>>> > ===
>>> > What should happen to JIRA tickets still targeting 2.4.4?
>>> > ===
>>> >
>>> > The current list of open tickets targeted at 2.4.4 can be found at:
>>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.4
>>> >
>>> > Committers should look at those and triage. Extremely important bug
>>> > fixes, documentation, and API tweaks that impact compatibility should
>>> > be worked on immediately. Everything else please retarget to an
>>> > appropriate release.
>>> >
>>> > ==
>>> > But my bug isn't fixed?
>>> > ==
>>> >
>>> > In order to make timely releases, we will typically not hold the
>>> > release unless the bug in question is a regression from the previous
>>> > release. That being said, if there is something which is a regression
>>> > that has not been correctly targeted please ping me or a committer to
>>> > help target the issue.
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

[Felix Cheung]

+1

Run tests, R tests, r-hub Debian, Ubuntu, mac, Windows


From: Hyukjin Kwon 
Sent: Wednesday, August 28, 2019 9:14 PM
To: Takeshi Yamamuro
Cc: dev; Dongjoon Hyun
Subject: Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1 (from the last blocker PR)

2019년 8월 29일 (목) 오전 8:20, Takeshi Yamamuro 
mailto:linguin@gmail.com>>님이 작성:
I checked the tests passed again on the same env.
It looks ok.


On Thu, Aug 29, 2019 at 6:15 AM Marcelo Vanzin  
wrote:
+1

On Tue, Aug 27, 2019 at 4:06 PM Dongjoon Hyun 
mailto:dongjoon.h...@gmail.com>> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 
> 2.4.4.
>
> The vote is open until August 30th 5PM PST and passes if a majority +1 PMC 
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.4
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.4-rc3 (commit 
> 7955b3962ac46b89564e0613db7bea98a1478bf2):
> https://github.com/apache/spark/tree/v2.4.4-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1332/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/
>
> The list of bug fixes going into 2.4.4 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12345466
>
> This release is using the release script of the tag v2.4.4-rc3.
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.4?
> ===
>
> The current list of open tickets targeted at 2.4.4 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target 
> Version/s" = 2.4.4
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.



--
Marcelo

-
To unsubscribe e-mail: 
dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org>



--
---
Takeshi Yamamuro

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

[Hyukjin Kwon]

+1 (from the last blocker PR)

2019년 8월 29일 (목) 오전 8:20, Takeshi Yamamuro 님이 작성:

> I checked the tests passed again on the same env.
> It looks ok.
>
>
> On Thu, Aug 29, 2019 at 6:15 AM Marcelo Vanzin 
> wrote:
>
>> +1
>>
>> On Tue, Aug 27, 2019 at 4:06 PM Dongjoon Hyun 
>> wrote:
>> >
>> > Please vote on releasing the following candidate as Apache Spark
>> version 2.4.4.
>> >
>> > The vote is open until August 30th 5PM PST and passes if a majority +1
>> PMC votes are cast, with a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Release this package as Apache Spark 2.4.4
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see http://spark.apache.org/
>> >
>> > The tag to be voted on is v2.4.4-rc3 (commit
>> 7955b3962ac46b89564e0613db7bea98a1478bf2):
>> > https://github.com/apache/spark/tree/v2.4.4-rc3
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/
>> >
>> > Signatures used for Spark RCs can be found in this file:
>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>> >
>> > The staging repository for this release can be found at:
>> > https://repository.apache.org/content/repositories/orgapachespark-1332/
>> >
>> > The documentation corresponding to this release can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/
>> >
>> > The list of bug fixes going into 2.4.4 can be found at the following
>> URL:
>> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
>> >
>> > This release is using the release script of the tag v2.4.4-rc3.
>> >
>> > FAQ
>> >
>> > =
>> > How can I help test this release?
>> > =
>> >
>> > If you are a Spark user, you can help us test this release by taking
>> > an existing Spark workload and running on this release candidate, then
>> > reporting any regressions.
>> >
>> > If you're working in PySpark you can set up a virtual env and install
>> > the current RC and see if anything important breaks, in the Java/Scala
>> > you can add the staging repository to your projects resolvers and test
>> > with the RC (make sure to clean up the artifact cache before/after so
>> > you don't end up building with a out of date RC going forward).
>> >
>> > ===
>> > What should happen to JIRA tickets still targeting 2.4.4?
>> > ===
>> >
>> > The current list of open tickets targeted at 2.4.4 can be found at:
>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.4
>> >
>> > Committers should look at those and triage. Extremely important bug
>> > fixes, documentation, and API tweaks that impact compatibility should
>> > be worked on immediately. Everything else please retarget to an
>> > appropriate release.
>> >
>> > ==
>> > But my bug isn't fixed?
>> > ==
>> >
>> > In order to make timely releases, we will typically not hold the
>> > release unless the bug in question is a regression from the previous
>> > release. That being said, if there is something which is a regression
>> > that has not been correctly targeted please ping me or a committer to
>> > help target the issue.
>>
>>
>>
>> --
>> Marcelo
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>
> --
> ---
> Takeshi Yamamuro
>

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

[Takeshi Yamamuro]

I checked the tests passed again on the same env.
It looks ok.


On Thu, Aug 29, 2019 at 6:15 AM Marcelo Vanzin 
wrote:

> +1
>
> On Tue, Aug 27, 2019 at 4:06 PM Dongjoon Hyun 
> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version
> 2.4.4.
> >
> > The vote is open until August 30th 5PM PST and passes if a majority +1
> PMC votes are cast, with a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.4
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.4-rc3 (commit
> 7955b3962ac46b89564e0613db7bea98a1478bf2):
> > https://github.com/apache/spark/tree/v2.4.4-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1332/
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/
> >
> > The list of bug fixes going into 2.4.4 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
> >
> > This release is using the release script of the tag v2.4.4-rc3.
> >
> > FAQ
> >
> > =
> > How can I help test this release?
> > =
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===
> > What should happen to JIRA tickets still targeting 2.4.4?
> > ===
> >
> > The current list of open tickets targeted at 2.4.4 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.4
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==
> > But my bug isn't fixed?
> > ==
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.
>
>
>
> --
> Marcelo
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

-- 
---
Takeshi Yamamuro

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

[Marcelo Vanzin]

+1

On Tue, Aug 27, 2019 at 4:06 PM Dongjoon Hyun  wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 
> 2.4.4.
>
> The vote is open until August 30th 5PM PST and passes if a majority +1 PMC 
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.4
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.4-rc3 (commit 
> 7955b3962ac46b89564e0613db7bea98a1478bf2):
> https://github.com/apache/spark/tree/v2.4.4-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1332/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/
>
> The list of bug fixes going into 2.4.4 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12345466
>
> This release is using the release script of the tag v2.4.4-rc3.
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.4?
> ===
>
> The current list of open tickets targeted at 2.4.4 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target 
> Version/s" = 2.4.4
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.



-- 
Marcelo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

[Holden Karau]

+1
Installed PySpark in a py3 virtual env & checked the PKG-INFO which we've
had difficulty with previously and looked good.

On Wed, Aug 28, 2019 at 10:17 AM DB Tsai  wrote:

> +1
>
> Thanks!
>
> On Wed, Aug 28, 2019 at 7:14 AM Wenchen Fan  wrote:
>
>> +1, no more blocking issues that I'm aware of.
>>
>> On Wed, Aug 28, 2019 at 8:33 PM Sean Owen  wrote:
>>
>>> +1 from me again.
>>>
>>> On Tue, Aug 27, 2019 at 6:06 PM Dongjoon Hyun 
>>> wrote:
>>> >
>>> > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.4.
>>> >
>>> > The vote is open until August 30th 5PM PST and passes if a majority +1
>>> PMC votes are cast, with a minimum of 3 +1 votes.
>>> >
>>> > [ ] +1 Release this package as Apache Spark 2.4.4
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > To learn more about Apache Spark, please see http://spark.apache.org/
>>> >
>>> > The tag to be voted on is v2.4.4-rc3 (commit
>>> 7955b3962ac46b89564e0613db7bea98a1478bf2):
>>> > https://github.com/apache/spark/tree/v2.4.4-rc3
>>> >
>>> > The release files, including signatures, digests, etc. can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/
>>> >
>>> > Signatures used for Spark RCs can be found in this file:
>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> >
>>> > The staging repository for this release can be found at:
>>> >
>>> https://repository.apache.org/content/repositories/orgapachespark-1332/
>>> >
>>> > The documentation corresponding to this release can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/
>>> >
>>> > The list of bug fixes going into 2.4.4 can be found at the following
>>> URL:
>>> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
>>> >
>>> > This release is using the release script of the tag v2.4.4-rc3.
>>> >
>>> > FAQ
>>> >
>>> > =
>>> > How can I help test this release?
>>> > =
>>> >
>>> > If you are a Spark user, you can help us test this release by taking
>>> > an existing Spark workload and running on this release candidate, then
>>> > reporting any regressions.
>>> >
>>> > If you're working in PySpark you can set up a virtual env and install
>>> > the current RC and see if anything important breaks, in the Java/Scala
>>> > you can add the staging repository to your projects resolvers and test
>>> > with the RC (make sure to clean up the artifact cache before/after so
>>> > you don't end up building with a out of date RC going forward).
>>> >
>>> > ===
>>> > What should happen to JIRA tickets still targeting 2.4.4?
>>> > ===
>>> >
>>> > The current list of open tickets targeted at 2.4.4 can be found at:
>>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.4
>>> >
>>> > Committers should look at those and triage. Extremely important bug
>>> > fixes, documentation, and API tweaks that impact compatibility should
>>> > be worked on immediately. Everything else please retarget to an
>>> > appropriate release.
>>> >
>>> > ==
>>> > But my bug isn't fixed?
>>> > ==
>>> >
>>> > In order to make timely releases, we will typically not hold the
>>> > release unless the bug in question is a regression from the previous
>>> > release. That being said, if there is something which is a regression
>>> > that has not been correctly targeted please ping me or a committer to
>>> > help target the issue.
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>> --
> - DB Sent from my iPhone
>


-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

[DB Tsai]

+1

Thanks!

On Wed, Aug 28, 2019 at 7:14 AM Wenchen Fan  wrote:

> +1, no more blocking issues that I'm aware of.
>
> On Wed, Aug 28, 2019 at 8:33 PM Sean Owen  wrote:
>
>> +1 from me again.
>>
>> On Tue, Aug 27, 2019 at 6:06 PM Dongjoon Hyun 
>> wrote:
>> >
>> > Please vote on releasing the following candidate as Apache Spark
>> version 2.4.4.
>> >
>> > The vote is open until August 30th 5PM PST and passes if a majority +1
>> PMC votes are cast, with a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Release this package as Apache Spark 2.4.4
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see http://spark.apache.org/
>> >
>> > The tag to be voted on is v2.4.4-rc3 (commit
>> 7955b3962ac46b89564e0613db7bea98a1478bf2):
>> > https://github.com/apache/spark/tree/v2.4.4-rc3
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/
>> >
>> > Signatures used for Spark RCs can be found in this file:
>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>> >
>> > The staging repository for this release can be found at:
>> > https://repository.apache.org/content/repositories/orgapachespark-1332/
>> >
>> > The documentation corresponding to this release can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/
>> >
>> > The list of bug fixes going into 2.4.4 can be found at the following
>> URL:
>> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
>> >
>> > This release is using the release script of the tag v2.4.4-rc3.
>> >
>> > FAQ
>> >
>> > =
>> > How can I help test this release?
>> > =
>> >
>> > If you are a Spark user, you can help us test this release by taking
>> > an existing Spark workload and running on this release candidate, then
>> > reporting any regressions.
>> >
>> > If you're working in PySpark you can set up a virtual env and install
>> > the current RC and see if anything important breaks, in the Java/Scala
>> > you can add the staging repository to your projects resolvers and test
>> > with the RC (make sure to clean up the artifact cache before/after so
>> > you don't end up building with a out of date RC going forward).
>> >
>> > ===
>> > What should happen to JIRA tickets still targeting 2.4.4?
>> > ===
>> >
>> > The current list of open tickets targeted at 2.4.4 can be found at:
>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.4
>> >
>> > Committers should look at those and triage. Extremely important bug
>> > fixes, documentation, and API tweaks that impact compatibility should
>> > be worked on immediately. Everything else please retarget to an
>> > appropriate release.
>> >
>> > ==
>> > But my bug isn't fixed?
>> > ==
>> >
>> > In order to make timely releases, we will typically not hold the
>> > release unless the bug in question is a regression from the previous
>> > release. That being said, if there is something which is a regression
>> > that has not been correctly targeted please ping me or a committer to
>> > help target the issue.
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>> --
- DB Sent from my iPhone

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

[Wenchen Fan]

+1, no more blocking issues that I'm aware of.

On Wed, Aug 28, 2019 at 8:33 PM Sean Owen  wrote:

> +1 from me again.
>
> On Tue, Aug 27, 2019 at 6:06 PM Dongjoon Hyun 
> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version
> 2.4.4.
> >
> > The vote is open until August 30th 5PM PST and passes if a majority +1
> PMC votes are cast, with a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.4
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.4-rc3 (commit
> 7955b3962ac46b89564e0613db7bea98a1478bf2):
> > https://github.com/apache/spark/tree/v2.4.4-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1332/
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/
> >
> > The list of bug fixes going into 2.4.4 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
> >
> > This release is using the release script of the tag v2.4.4-rc3.
> >
> > FAQ
> >
> > =
> > How can I help test this release?
> > =
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===
> > What should happen to JIRA tickets still targeting 2.4.4?
> > ===
> >
> > The current list of open tickets targeted at 2.4.4 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.4
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==
> > But my bug isn't fixed?
> > ==
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

[Sean Owen]

+1 from me again.

On Tue, Aug 27, 2019 at 6:06 PM Dongjoon Hyun  wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 
> 2.4.4.
>
> The vote is open until August 30th 5PM PST and passes if a majority +1 PMC 
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.4
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.4-rc3 (commit 
> 7955b3962ac46b89564e0613db7bea98a1478bf2):
> https://github.com/apache/spark/tree/v2.4.4-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1332/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/
>
> The list of bug fixes going into 2.4.4 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12345466
>
> This release is using the release script of the tag v2.4.4-rc3.
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.4?
> ===
>
> The current list of open tickets targeted at 2.4.4 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target 
> Version/s" = 2.4.4
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

RE: [VOTE] Release Apache Spark 2.4.4 (RC3)

[Kazuaki Ishizaki]

+1
Built and tested with `mvn -Pyarn -Phadoop-2.7 -Pkubernetes -Pkinesis-asl 
-Phive -Phive-thriftserver test` on OpenJDK 1.8.0_211 on Ubuntu 16.04 
x86_64

Regards,
Kazuaki Ishizaki



From:   Dongjoon Hyun 
To: dev 
Date:   2019/08/28 12:14
Subject:[EXTERNAL] Re: [VOTE] Release Apache Spark 2.4.4 (RC3)



+1.

- Checked checksums and signatures of artifacts.
- Checked to have all binaries and maven repo.
- Checked document generation (including a new change after RC2)
- Build with `-Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver 
-Phadoop-2.6` on AdoptOpenJDK8_202.
- Tested with both Scala-2.11/Scala-2.12 and both Python2/3.
   Python 2.7.15 with numpy 1.16.4, scipy 1.2.2, pandas 0.19.2, pyarrow 
0.8.0
   Python 3.6.4 with numpy 1.16.4, scipy 1.2.2, pandas 0.23.2, pyarrow 
0.11.0
- Tested JDBC IT.

Bests,
Dongjoon.


On Tue, Aug 27, 2019 at 4:05 PM Dongjoon Hyun  
wrote:
Please vote on releasing the following candidate as Apache Spark version 
2.4.4.

The vote is open until August 30th 5PM PST and passes if a majority +1 PMC 
votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.4
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.4-rc3 (commit 
7955b3962ac46b89564e0613db7bea98a1478bf2):
https://github.com/apache/spark/tree/v2.4.4-rc3

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1332/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/

The list of bug fixes going into 2.4.4 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12345466

This release is using the release script of the tag v2.4.4-rc3.

FAQ

=
How can I help test this release?
=

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===
What should happen to JIRA tickets still targeting 2.4.4?
===

The current list of open tickets targeted at 2.4.4 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target 
Version/s" = 2.4.4

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==
But my bug isn't fixed?
==

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

[Dongjoon Hyun]

+1.

- Checked checksums and signatures of artifacts.
- Checked to have all binaries and maven repo.
- Checked document generation (including a new change after RC2)
- Build with `-Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver
-Phadoop-2.6` on AdoptOpenJDK8_202.
- Tested with both Scala-2.11/Scala-2.12 and both Python2/3.
   Python 2.7.15 with numpy 1.16.4, scipy 1.2.2, pandas 0.19.2, pyarrow
0.8.0
   Python 3.6.4 with numpy 1.16.4, scipy 1.2.2, pandas 0.23.2, pyarrow
0.11.0
- Tested JDBC IT.

Bests,
Dongjoon.


On Tue, Aug 27, 2019 at 4:05 PM Dongjoon Hyun 
wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 2.4.4.
>
> The vote is open until August 30th 5PM PST and passes if a majority +1 PMC
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.4
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.4-rc3 (commit
> 7955b3962ac46b89564e0613db7bea98a1478bf2):
> https://github.com/apache/spark/tree/v2.4.4-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1332/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/
>
> The list of bug fixes going into 2.4.4 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12345466
>
> This release is using the release script of the tag v2.4.4-rc3.
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.4?
> ===
>
> The current list of open tickets targeted at 2.4.4 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.4
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>

[VOTE] Release Apache Spark 2.4.4 (RC3)

[Dongjoon Hyun]

Please vote on releasing the following candidate as Apache Spark version
2.4.4.

The vote is open until August 30th 5PM PST and passes if a majority +1 PMC
votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.4
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.4-rc3 (commit
7955b3962ac46b89564e0613db7bea98a1478bf2):
https://github.com/apache/spark/tree/v2.4.4-rc3

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1332/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc3-docs/

The list of bug fixes going into 2.4.4 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12345466

This release is using the release script of the tag v2.4.4-rc3.

FAQ

=
How can I help test this release?
=

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===
What should happen to JIRA tickets still targeting 2.4.4?
===

The current list of open tickets targeted at 2.4.4 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target
Version/s" = 2.4.4

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==
But my bug isn't fixed?
==

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Re: [VOTE] Release Apache Spark 2.4.4 (RC2)

[Dongjoon Hyun]

Hi, All.

Thank you for testing.

2.4.4 RC2 vote fails due to the PySpark correctness issue.
Since the blocker is merged, I'll start RC3 soon.

Bests,
Dongjoon.


On Mon, Aug 26, 2019 at 10:55 PM Hyukjin Kwon  wrote:

> -1
>
> Seems there's one critical correctness issue specifically in branch-2.4 ...
> Please take a look for https://github.com/apache/spark/pull/25593
>
> 2019년 8월 27일 (화) 오후 2:38, Takeshi Yamamuro 님이 작성:
>
>> Hi, Dongjoon
>>
>> I checked that all the test passed on my Mac/x86_64 env with:
>> -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
>> -Pkubernetes-integration-tests -Psparkr
>>
>> maropu@~/spark-2.4.4-rc2:$java -version
>> java version "1.8.0_181"
>> Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
>> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
>>
>> Bests,
>> Takeshi
>>
>>
>> On Tue, Aug 27, 2019 at 11:06 AM Sean Owen  wrote:
>>
>>> +1 as per response to RC1. The existing issues identified there seem
>>> to have been fixed.
>>>
>>>
>>> On Mon, Aug 26, 2019 at 2:45 AM Dongjoon Hyun 
>>> wrote:
>>> >
>>> > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.4.
>>> >
>>> > The vote is open until August 29th 1AM PST and passes if a majority +1
>>> PMC votes are cast, with a minimum of 3 +1 votes.
>>> >
>>> > [ ] +1 Release this package as Apache Spark 2.4.4
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > To learn more about Apache Spark, please see http://spark.apache.org/
>>> >
>>> > The tag to be voted on is v2.4.4-rc2 (commit
>>> b7a15b69aca8a2fc3f308105e5978a69dff0f4fb):
>>> > https://github.com/apache/spark/tree/v2.4.4-rc2
>>> >
>>> > The release files, including signatures, digests, etc. can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc2-bin/
>>> >
>>> > Signatures used for Spark RCs can be found in this file:
>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> >
>>> > The staging repository for this release can be found at:
>>> >
>>> https://repository.apache.org/content/repositories/orgapachespark-1327/
>>> >
>>> > The documentation corresponding to this release can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc2-docs/
>>> >
>>> > The list of bug fixes going into 2.4.4 can be found at the following
>>> URL:
>>> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
>>> >
>>> > This release is using the release script of the tag v2.4.4-rc2.
>>> >
>>> > FAQ
>>> >
>>> > =
>>> > How can I help test this release?
>>> > =
>>> >
>>> > If you are a Spark user, you can help us test this release by taking
>>> > an existing Spark workload and running on this release candidate, then
>>> > reporting any regressions.
>>> >
>>> > If you're working in PySpark you can set up a virtual env and install
>>> > the current RC and see if anything important breaks, in the Java/Scala
>>> > you can add the staging repository to your projects resolvers and test
>>> > with the RC (make sure to clean up the artifact cache before/after so
>>> > you don't end up building with a out of date RC going forward).
>>> >
>>> > ===
>>> > What should happen to JIRA tickets still targeting 2.4.4?
>>> > ===
>>> >
>>> > The current list of open tickets targeted at 2.4.4 can be found at:
>>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.4
>>> >
>>> > Committers should look at those and triage. Extremely important bug
>>> > fixes, documentation, and API tweaks that impact compatibility should
>>> > be worked on immediately. Everything else please retarget to an
>>> > appropriate release.
>>> >
>>> > ==
>>> > But my bug isn't fixed?
>>> > ==
>>> >
>>> > In order to make timely releases, we will typically not hold the
>>> > release unless the bug in question is a regression from the previous
>>> > release. That being said, if there is something which is a regression
>>> > that has not been correctly targeted please ping me or a committer to
>>> > help target the issue.
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>

Re: [VOTE] Release Apache Spark 2.4.4 (RC2)

[Hyukjin Kwon]

-1

Seems there's one critical correctness issue specifically in branch-2.4 ...
Please take a look for https://github.com/apache/spark/pull/25593

2019년 8월 27일 (화) 오후 2:38, Takeshi Yamamuro 님이 작성:

> Hi, Dongjoon
>
> I checked that all the test passed on my Mac/x86_64 env with:
> -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
> -Pkubernetes-integration-tests -Psparkr
>
> maropu@~/spark-2.4.4-rc2:$java -version
> java version "1.8.0_181"
> Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
>
> Bests,
> Takeshi
>
>
> On Tue, Aug 27, 2019 at 11:06 AM Sean Owen  wrote:
>
>> +1 as per response to RC1. The existing issues identified there seem
>> to have been fixed.
>>
>>
>> On Mon, Aug 26, 2019 at 2:45 AM Dongjoon Hyun 
>> wrote:
>> >
>> > Please vote on releasing the following candidate as Apache Spark
>> version 2.4.4.
>> >
>> > The vote is open until August 29th 1AM PST and passes if a majority +1
>> PMC votes are cast, with a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Release this package as Apache Spark 2.4.4
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see http://spark.apache.org/
>> >
>> > The tag to be voted on is v2.4.4-rc2 (commit
>> b7a15b69aca8a2fc3f308105e5978a69dff0f4fb):
>> > https://github.com/apache/spark/tree/v2.4.4-rc2
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc2-bin/
>> >
>> > Signatures used for Spark RCs can be found in this file:
>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>> >
>> > The staging repository for this release can be found at:
>> > https://repository.apache.org/content/repositories/orgapachespark-1327/
>> >
>> > The documentation corresponding to this release can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc2-docs/
>> >
>> > The list of bug fixes going into 2.4.4 can be found at the following
>> URL:
>> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
>> >
>> > This release is using the release script of the tag v2.4.4-rc2.
>> >
>> > FAQ
>> >
>> > =
>> > How can I help test this release?
>> > =
>> >
>> > If you are a Spark user, you can help us test this release by taking
>> > an existing Spark workload and running on this release candidate, then
>> > reporting any regressions.
>> >
>> > If you're working in PySpark you can set up a virtual env and install
>> > the current RC and see if anything important breaks, in the Java/Scala
>> > you can add the staging repository to your projects resolvers and test
>> > with the RC (make sure to clean up the artifact cache before/after so
>> > you don't end up building with a out of date RC going forward).
>> >
>> > ===
>> > What should happen to JIRA tickets still targeting 2.4.4?
>> > ===
>> >
>> > The current list of open tickets targeted at 2.4.4 can be found at:
>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.4
>> >
>> > Committers should look at those and triage. Extremely important bug
>> > fixes, documentation, and API tweaks that impact compatibility should
>> > be worked on immediately. Everything else please retarget to an
>> > appropriate release.
>> >
>> > ==
>> > But my bug isn't fixed?
>> > ==
>> >
>> > In order to make timely releases, we will typically not hold the
>> > release unless the bug in question is a regression from the previous
>> > release. That being said, if there is something which is a regression
>> > that has not been correctly targeted please ping me or a committer to
>> > help target the issue.
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>
> --
> ---
> Takeshi Yamamuro
>

Re: [VOTE] Release Apache Spark 2.4.4 (RC2)

[Takeshi Yamamuro]

Hi, Dongjoon

I checked that all the test passed on my Mac/x86_64 env with:
-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
-Pkubernetes-integration-tests -Psparkr

maropu@~/spark-2.4.4-rc2:$java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

Bests,
Takeshi


On Tue, Aug 27, 2019 at 11:06 AM Sean Owen  wrote:

> +1 as per response to RC1. The existing issues identified there seem
> to have been fixed.
>
>
> On Mon, Aug 26, 2019 at 2:45 AM Dongjoon Hyun 
> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version
> 2.4.4.
> >
> > The vote is open until August 29th 1AM PST and passes if a majority +1
> PMC votes are cast, with a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.4
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.4-rc2 (commit
> b7a15b69aca8a2fc3f308105e5978a69dff0f4fb):
> > https://github.com/apache/spark/tree/v2.4.4-rc2
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc2-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1327/
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc2-docs/
> >
> > The list of bug fixes going into 2.4.4 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
> >
> > This release is using the release script of the tag v2.4.4-rc2.
> >
> > FAQ
> >
> > =
> > How can I help test this release?
> > =
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===
> > What should happen to JIRA tickets still targeting 2.4.4?
> > ===
> >
> > The current list of open tickets targeted at 2.4.4 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.4
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==
> > But my bug isn't fixed?
> > ==
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

-- 
---
Takeshi Yamamuro

Re: [VOTE] Release Apache Spark 2.4.4 (RC2)

[Sean Owen]

+1 as per response to RC1. The existing issues identified there seem
to have been fixed.


On Mon, Aug 26, 2019 at 2:45 AM Dongjoon Hyun  wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 
> 2.4.4.
>
> The vote is open until August 29th 1AM PST and passes if a majority +1 PMC 
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.4
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.4-rc2 (commit 
> b7a15b69aca8a2fc3f308105e5978a69dff0f4fb):
> https://github.com/apache/spark/tree/v2.4.4-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1327/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc2-docs/
>
> The list of bug fixes going into 2.4.4 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12345466
>
> This release is using the release script of the tag v2.4.4-rc2.
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.4?
> ===
>
> The current list of open tickets targeted at 2.4.4 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target 
> Version/s" = 2.4.4
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[VOTE] Release Apache Spark 2.4.4 (RC2)

[Dongjoon Hyun]

Please vote on releasing the following candidate as Apache Spark version
2.4.4.

The vote is open until August 29th 1AM PST and passes if a majority +1 PMC
votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.4
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.4-rc2 (commit
b7a15b69aca8a2fc3f308105e5978a69dff0f4fb):
https://github.com/apache/spark/tree/v2.4.4-rc2

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc2-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1327/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc2-docs/

The list of bug fixes going into 2.4.4 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12345466

This release is using the release script of the tag v2.4.4-rc2.

FAQ

=
How can I help test this release?
=

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===
What should happen to JIRA tickets still targeting 2.4.4?
===

The current list of open tickets targeted at 2.4.4 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target
Version/s" = 2.4.4

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==
But my bug isn't fixed?
==

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

[Dongjoon Hyun]

FYI, three more patches landed to `branch-2.4` until yesterday.

[SPARK-28642][SQL][2.4] Hide credentials in show create table
[SPARK-27330][SS][2.4] support task abort in foreach writer
[SPARK-28025][SS][2.4] Fix FileContextBasedCheckpointFileManager leaking
crc files


For some non-blocker issues like `[SPARK-28778][MESOS] Fixed executors
advertised address ...`,
we can have it if it lands before `2.4.4-rc2` tag creation. I'll make
`2.4.4-rc2` tag tomorrow.

Please let me know if there is blocker issues.

Bests,
Dongjoon.


On Thu, Aug 22, 2019 at 9:28 AM Dongjoon Hyun 
wrote:

> Hi, All.
>
> This 2.4.4 RC1 vote fails and the reported PRs are merged to `branch-2.4`.
> The following is the commit list since `v2.4.4-rc1` tag.
>
> Preparing development version 2.4.5-SNAPSHOT
> [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8
> [SPARK-28775][CORE][TESTS] Skip date 8633 in Kwajalein due to changes in
> tzdata2018i that only some JDK 8s use
> [SPARK-28777][PYTHON][DOCS] Fix format_string doc string with the correct
> parameters
> [SPARK-28650][SS][DOC] Correct explanation of guarantee for ForeachWriter
> [SPARK-26895][CORE][2.4] prepareSubmitEnvironment should be called within
> doAs for proxy users
> [SPARK-28699][SQL] Disable using radix sort for ShuffleExchangeExec in
> repartition case
> [SPARK-28844][SQL] Fix typo in SQLConf FILE_COMRESSION_FACTOR
> [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
> [SPARK-28699][CORE][2.4] Fix a corner case for aborting indeterminate stage
>
>
> Please let me know if we need more patches.
> I'm going to cut `2.4.4-rc2` tag during weekend and starts RC2 on next
> Monday.
>
> Bests,
> Dongjoon.
>
>
> On Tue, Aug 20, 2019 at 5:01 AM Sean Owen  wrote:
>
>> Sounds fine, we probably needed SPARK-28775 anyway. I merged that and
>> SPARK-28749. It looks like it's just the one you're talking about
>> right now, SPARK-28699.
>> The rest of the tests seemed to pass OK, release looks good, but bears
>> more testing by everyone out there before a next RC.
>>
>> On Tue, Aug 20, 2019 at 12:10 AM Wenchen Fan  wrote:
>> >
>> > Unfortunately, I need to -1.
>> >
>> > Recently we found that the repartition correctness bug can still be
>> reproduced. The root cause has been identified and there are 2 PRs to fix 2
>> related issues:
>> > https://github.com/apache/spark/pull/25491
>> > https://github.com/apache/spark/pull/25498
>> >
>> > I think we should have this fix in 2.3 and 2.4.
>> >
>> > Thanks,
>> > Wenchen
>> >
>> > On Tue, Aug 20, 2019 at 7:32 AM Dongjoon Hyun 
>> wrote:
>> >>
>> >> Thank you for testing, Sean and Herman.
>> >>
>> >> There are three reporting until now.
>> >>
>> >> 1. SPARK-28775 is for JDK 8u221+ testing at Apache Spark 3.0/2.4/2.3.
>> >> 2. SPARK-28749 is for Scala 2.12 Python testing at Apache Spark 2.4
>> only.
>> >> 3. SPARK-28699 is for disabling radix sort for ShuffleExchangeExec at
>> Apache Spark 3.0/2.4/2.3.
>> >>
>> >> Both (1) and (2) are nice-to-have and test-only fixes. (3) could be a
>> correctness issue, but it seems that there are some other approaches.
>> >> I'm monitoring all reports. Let's see. For now, I'd like to continue
>> 2.4.4 RC1 voting for more testing.
>> >>
>> >> Bests,
>> >> Dongjoon.
>>
>

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

[Dongjoon Hyun]

Hi, All.

This 2.4.4 RC1 vote fails and the reported PRs are merged to `branch-2.4`.
The following is the commit list since `v2.4.4-rc1` tag.

Preparing development version 2.4.5-SNAPSHOT
[SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8
[SPARK-28775][CORE][TESTS] Skip date 8633 in Kwajalein due to changes in
tzdata2018i that only some JDK 8s use
[SPARK-28777][PYTHON][DOCS] Fix format_string doc string with the correct
parameters
[SPARK-28650][SS][DOC] Correct explanation of guarantee for ForeachWriter
[SPARK-26895][CORE][2.4] prepareSubmitEnvironment should be called within
doAs for proxy users
[SPARK-28699][SQL] Disable using radix sort for ShuffleExchangeExec in
repartition case
[SPARK-28844][SQL] Fix typo in SQLConf FILE_COMRESSION_FACTOR
[SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
[SPARK-28699][CORE][2.4] Fix a corner case for aborting indeterminate stage


Please let me know if we need more patches.
I'm going to cut `2.4.4-rc2` tag during weekend and starts RC2 on next
Monday.

Bests,
Dongjoon.


On Tue, Aug 20, 2019 at 5:01 AM Sean Owen  wrote:

> Sounds fine, we probably needed SPARK-28775 anyway. I merged that and
> SPARK-28749. It looks like it's just the one you're talking about
> right now, SPARK-28699.
> The rest of the tests seemed to pass OK, release looks good, but bears
> more testing by everyone out there before a next RC.
>
> On Tue, Aug 20, 2019 at 12:10 AM Wenchen Fan  wrote:
> >
> > Unfortunately, I need to -1.
> >
> > Recently we found that the repartition correctness bug can still be
> reproduced. The root cause has been identified and there are 2 PRs to fix 2
> related issues:
> > https://github.com/apache/spark/pull/25491
> > https://github.com/apache/spark/pull/25498
> >
> > I think we should have this fix in 2.3 and 2.4.
> >
> > Thanks,
> > Wenchen
> >
> > On Tue, Aug 20, 2019 at 7:32 AM Dongjoon Hyun 
> wrote:
> >>
> >> Thank you for testing, Sean and Herman.
> >>
> >> There are three reporting until now.
> >>
> >> 1. SPARK-28775 is for JDK 8u221+ testing at Apache Spark 3.0/2.4/2.3.
> >> 2. SPARK-28749 is for Scala 2.12 Python testing at Apache Spark 2.4
> only.
> >> 3. SPARK-28699 is for disabling radix sort for ShuffleExchangeExec at
> Apache Spark 3.0/2.4/2.3.
> >>
> >> Both (1) and (2) are nice-to-have and test-only fixes. (3) could be a
> correctness issue, but it seems that there are some other approaches.
> >> I'm monitoring all reports. Let's see. For now, I'd like to continue
> 2.4.4 RC1 voting for more testing.
> >>
> >> Bests,
> >> Dongjoon.
>

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

[Sean Owen]

Sounds fine, we probably needed SPARK-28775 anyway. I merged that and
SPARK-28749. It looks like it's just the one you're talking about
right now, SPARK-28699.
The rest of the tests seemed to pass OK, release looks good, but bears
more testing by everyone out there before a next RC.

On Tue, Aug 20, 2019 at 12:10 AM Wenchen Fan  wrote:
>
> Unfortunately, I need to -1.
>
> Recently we found that the repartition correctness bug can still be 
> reproduced. The root cause has been identified and there are 2 PRs to fix 2 
> related issues:
> https://github.com/apache/spark/pull/25491
> https://github.com/apache/spark/pull/25498
>
> I think we should have this fix in 2.3 and 2.4.
>
> Thanks,
> Wenchen
>
> On Tue, Aug 20, 2019 at 7:32 AM Dongjoon Hyun  wrote:
>>
>> Thank you for testing, Sean and Herman.
>>
>> There are three reporting until now.
>>
>> 1. SPARK-28775 is for JDK 8u221+ testing at Apache Spark 3.0/2.4/2.3.
>> 2. SPARK-28749 is for Scala 2.12 Python testing at Apache Spark 2.4 only.
>> 3. SPARK-28699 is for disabling radix sort for ShuffleExchangeExec at Apache 
>> Spark 3.0/2.4/2.3.
>>
>> Both (1) and (2) are nice-to-have and test-only fixes. (3) could be a 
>> correctness issue, but it seems that there are some other approaches.
>> I'm monitoring all reports. Let's see. For now, I'd like to continue 2.4.4 
>> RC1 voting for more testing.
>>
>> Bests,
>> Dongjoon.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

[Wenchen Fan]

Unfortunately, I need to -1.

Recently we found that the repartition correctness bug can still be
reproduced. The root cause has been identified and there are 2 PRs to fix 2
related issues:
https://github.com/apache/spark/pull/25491
https://github.com/apache/spark/pull/25498

I think we should have this fix in 2.3 and 2.4.

Thanks,
Wenchen

On Tue, Aug 20, 2019 at 7:32 AM Dongjoon Hyun 
wrote:

> Thank you for testing, Sean and Herman.
>
> There are three reporting until now.
>
> 1. SPARK-28775 is for JDK 8u221+ testing at Apache Spark 3.0/2.4/2.3.
> 2. SPARK-28749 is for Scala 2.12 Python testing at Apache Spark 2.4 only.
> 3. SPARK-28699 is for disabling radix sort for ShuffleExchangeExec at
> Apache Spark 3.0/2.4/2.3.
>
> Both (1) and (2) are nice-to-have and test-only fixes. (3) could be a
> correctness issue, but it seems that there are some other approaches.
> I'm monitoring all reports. Let's see. For now, I'd like to continue 2.4.4
> RC1 voting for more testing.
>
> Bests,
> Dongjoon.
>
>
> On Mon, Aug 19, 2019 at 2:09 PM Herman van Hovell 
> wrote:
>
>> The error you are seeing is caused by
>> https://issues.apache.org/jira/browse/SPARK-28775.
>>
>>
>> On Mon, Aug 19, 2019 at 10:40 PM Sean Owen  wrote:
>>
>>> Things are looking pretty good so far, but a few notes:
>>>
>>> I thought we might need this PR to make the 2.12 build of 2.4.x not
>>> try to build Kafka 0.8 support, but, I'm not seeing that 2.4.x + 2.12
>>> builds or tests it?
>>> https://github.com/apache/spark/pull/25482
>>> I can merge this to 2.4 shortly anyway, but not clear it affects the RC.
>>>
>>>
>>> I'm getting one weird failure in tests:
>>>
>>> - daysToMillis and millisToDays *** FAILED ***
>>>   8634 did not equal 8633 Round trip of 8633 did not work in tz
>>>
>>> sun.util.calendar.ZoneInfo[id="Kwajalein",offset=4320,dstSavings=0,useDaylight=false,transitions=8,lastRule=null]
>>> (DateTimeUtilsSuite.scala:683)
>>>
>>> See
>>> https://github.com/apache/spark/pull/19234#pullrequestreview-64463435
>>> for some context and
>>>
>>> https://github.com/apache/spark/commit/c5b8d54c61780af6e9e157e6c855718df972efad
>>> for a fix for a similar type of issue.
>>>
>>> This may be quite specific to a particular version of Java 8, but I'm
>>> testing on the latest (1.8.0_222). We can 'patch' it by allowing for
>>> multiple correct answers here.
>>> It may not hold up the RC unless others see the failure, but I can
>>> work on that anyway.
>>>
>>> On Mon, Aug 19, 2019 at 11:55 AM Dongjoon Hyun 
>>> wrote:
>>> >
>>> > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.4.
>>> >
>>> > The vote is open until August 22nd 10AM PST and passes if a majority
>>> +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>> >
>>> > [ ] +1 Release this package as Apache Spark 2.4.4
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > To learn more about Apache Spark, please see http://spark.apache.org/
>>> >
>>> > The tag to be voted on is v2.4.4-rc1 (commit
>>> 13f2465c6c8328e988f7215ee5f5d2c5e69e8d21):
>>> > https://github.com/apache/spark/tree/v2.4.4-rc1
>>> >
>>> > The release files, including signatures, digests, etc. can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc1-bin/
>>> >
>>> > Signatures used for Spark RCs can be found in this file:
>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> >
>>> > The staging repository for this release can be found at:
>>> >
>>> https://repository.apache.org/content/repositories/orgapachespark-1326/
>>> >
>>> > The documentation corresponding to this release can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc1-docs/
>>> >
>>> > The list of bug fixes going into 2.4.4 can be found at the following
>>> URL:
>>> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
>>> >
>>> > This release is using the release script of the tag v2.4.4-rc1.
>>> >
>>> > FAQ
>>> >
>>> > =
>>> > How can I help test this release?
>>> > =
>>> >
>>> > If you are a Spark user, you can help us test this release by taking
>>> > an existing Spark workload and running on this release candidate, then
>>> > reporting any regressions.
>>> >
>>> > If you're working in PySpark you can set up a virtual env and install
>>> > the current RC and see if anything important breaks, in the Java/Scala
>>> > you can add the staging repository to your projects resolvers and test
>>> > with the RC (make sure to clean up the artifact cache before/after so
>>> > you don't end up building with a out of date RC going forward).
>>> >
>>> > ===
>>> > What should happen to JIRA tickets still targeting 2.4.4?
>>> > ===
>>> >
>>> > The current list of open tickets targeted at 2.4.4 can be found at:
>>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.4
>>> >
>>> > Committers should look 

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

[Dongjoon Hyun]

Thank you for testing, Sean and Herman.

There are three reporting until now.

1. SPARK-28775 is for JDK 8u221+ testing at Apache Spark 3.0/2.4/2.3.
2. SPARK-28749 is for Scala 2.12 Python testing at Apache Spark 2.4 only.
3. SPARK-28699 is for disabling radix sort for ShuffleExchangeExec at
Apache Spark 3.0/2.4/2.3.

Both (1) and (2) are nice-to-have and test-only fixes. (3) could be a
correctness issue, but it seems that there are some other approaches.
I'm monitoring all reports. Let's see. For now, I'd like to continue 2.4.4
RC1 voting for more testing.

Bests,
Dongjoon.


On Mon, Aug 19, 2019 at 2:09 PM Herman van Hovell 
wrote:

> The error you are seeing is caused by
> https://issues.apache.org/jira/browse/SPARK-28775.
>
>
> On Mon, Aug 19, 2019 at 10:40 PM Sean Owen  wrote:
>
>> Things are looking pretty good so far, but a few notes:
>>
>> I thought we might need this PR to make the 2.12 build of 2.4.x not
>> try to build Kafka 0.8 support, but, I'm not seeing that 2.4.x + 2.12
>> builds or tests it?
>> https://github.com/apache/spark/pull/25482
>> I can merge this to 2.4 shortly anyway, but not clear it affects the RC.
>>
>>
>> I'm getting one weird failure in tests:
>>
>> - daysToMillis and millisToDays *** FAILED ***
>>   8634 did not equal 8633 Round trip of 8633 did not work in tz
>>
>> sun.util.calendar.ZoneInfo[id="Kwajalein",offset=4320,dstSavings=0,useDaylight=false,transitions=8,lastRule=null]
>> (DateTimeUtilsSuite.scala:683)
>>
>> See https://github.com/apache/spark/pull/19234#pullrequestreview-64463435
>> for some context and
>>
>> https://github.com/apache/spark/commit/c5b8d54c61780af6e9e157e6c855718df972efad
>> for a fix for a similar type of issue.
>>
>> This may be quite specific to a particular version of Java 8, but I'm
>> testing on the latest (1.8.0_222). We can 'patch' it by allowing for
>> multiple correct answers here.
>> It may not hold up the RC unless others see the failure, but I can
>> work on that anyway.
>>
>> On Mon, Aug 19, 2019 at 11:55 AM Dongjoon Hyun 
>> wrote:
>> >
>> > Please vote on releasing the following candidate as Apache Spark
>> version 2.4.4.
>> >
>> > The vote is open until August 22nd 10AM PST and passes if a majority +1
>> PMC votes are cast, with a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Release this package as Apache Spark 2.4.4
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see http://spark.apache.org/
>> >
>> > The tag to be voted on is v2.4.4-rc1 (commit
>> 13f2465c6c8328e988f7215ee5f5d2c5e69e8d21):
>> > https://github.com/apache/spark/tree/v2.4.4-rc1
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc1-bin/
>> >
>> > Signatures used for Spark RCs can be found in this file:
>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>> >
>> > The staging repository for this release can be found at:
>> > https://repository.apache.org/content/repositories/orgapachespark-1326/
>> >
>> > The documentation corresponding to this release can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc1-docs/
>> >
>> > The list of bug fixes going into 2.4.4 can be found at the following
>> URL:
>> > https://issues.apache.org/jira/projects/SPARK/versions/12345466
>> >
>> > This release is using the release script of the tag v2.4.4-rc1.
>> >
>> > FAQ
>> >
>> > =
>> > How can I help test this release?
>> > =
>> >
>> > If you are a Spark user, you can help us test this release by taking
>> > an existing Spark workload and running on this release candidate, then
>> > reporting any regressions.
>> >
>> > If you're working in PySpark you can set up a virtual env and install
>> > the current RC and see if anything important breaks, in the Java/Scala
>> > you can add the staging repository to your projects resolvers and test
>> > with the RC (make sure to clean up the artifact cache before/after so
>> > you don't end up building with a out of date RC going forward).
>> >
>> > ===
>> > What should happen to JIRA tickets still targeting 2.4.4?
>> > ===
>> >
>> > The current list of open tickets targeted at 2.4.4 can be found at:
>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.4
>> >
>> > Committers should look at those and triage. Extremely important bug
>> > fixes, documentation, and API tweaks that impact compatibility should
>> > be worked on immediately. Everything else please retarget to an
>> > appropriate release.
>> >
>> > ==
>> > But my bug isn't fixed?
>> > ==
>> >
>> > In order to make timely releases, we will typically not hold the
>> > release unless the bug in question is a regression from the previous
>> > release. That being said, if there is something which is a regression
>> > that has 

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

[Sean Owen]

Things are looking pretty good so far, but a few notes:

I thought we might need this PR to make the 2.12 build of 2.4.x not
try to build Kafka 0.8 support, but, I'm not seeing that 2.4.x + 2.12
builds or tests it?
https://github.com/apache/spark/pull/25482
I can merge this to 2.4 shortly anyway, but not clear it affects the RC.


I'm getting one weird failure in tests:

- daysToMillis and millisToDays *** FAILED ***
  8634 did not equal 8633 Round trip of 8633 did not work in tz
sun.util.calendar.ZoneInfo[id="Kwajalein",offset=4320,dstSavings=0,useDaylight=false,transitions=8,lastRule=null]
(DateTimeUtilsSuite.scala:683)

See https://github.com/apache/spark/pull/19234#pullrequestreview-64463435
for some context and
https://github.com/apache/spark/commit/c5b8d54c61780af6e9e157e6c855718df972efad
for a fix for a similar type of issue.

This may be quite specific to a particular version of Java 8, but I'm
testing on the latest (1.8.0_222). We can 'patch' it by allowing for
multiple correct answers here.
It may not hold up the RC unless others see the failure, but I can
work on that anyway.

On Mon, Aug 19, 2019 at 11:55 AM Dongjoon Hyun  wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 
> 2.4.4.
>
> The vote is open until August 22nd 10AM PST and passes if a majority +1 PMC 
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.4
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.4-rc1 (commit 
> 13f2465c6c8328e988f7215ee5f5d2c5e69e8d21):
> https://github.com/apache/spark/tree/v2.4.4-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1326/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc1-docs/
>
> The list of bug fixes going into 2.4.4 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12345466
>
> This release is using the release script of the tag v2.4.4-rc1.
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.4?
> ===
>
> The current list of open tickets targeted at 2.4.4 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target 
> Version/s" = 2.4.4
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[VOTE] Release Apache Spark 2.4.4 (RC1)

[Dongjoon Hyun]

Please vote on releasing the following candidate as Apache Spark version
2.4.4.

The vote is open until August 22nd 10AM PST and passes if a majority +1 PMC
votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.4
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.4-rc1 (commit
13f2465c6c8328e988f7215ee5f5d2c5e69e8d21):
https://github.com/apache/spark/tree/v2.4.4-rc1

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc1-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1326/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.4-rc1-docs/

The list of bug fixes going into 2.4.4 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12345466

This release is using the release script of the tag v2.4.4-rc1.

FAQ

=
How can I help test this release?
=

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===
What should happen to JIRA tickets still targeting 2.4.4?
===

The current list of open tickets targeted at 2.4.4 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target
Version/s" = 2.4.4

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==
But my bug isn't fixed?
==

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Re: Release Apache Spark 2.4.4

[Dongjoon Hyun]

Thank you, Kazuaki.

Bests,
Dongjoon.

On Fri, Aug 16, 2019 at 3:10 AM Kazuaki Ishizaki 
wrote:

> Sure, I will launch a separate e-mail thread for discussing 2.3.4 later.
>
> Regards,
> Kazuaki Ishizaki, Ph.D.
>
>
>
> From:Dongjoon Hyun 
> To:Sean Owen , Kazuaki Ishizaki 
> Cc:dev 
> Date:2019/08/16 05:10
> Subject:    [EXTERNAL] Re: Release Apache Spark 2.4.4
> --
>
>
>
> +1 for that.
>
> Kazuaki volunteered for 2.3.4 release last month. AFAIK, he has been
> preparing that.
>
> -
> *https://lists.apache.org/thread.html/6fafeefb7715e8764ccfe5d30c90d7444378b5f4f383ec95e2f1d7de@%3Cdev.spark.apache.org%3E*
> <https://lists.apache.org/thread.html/6fafeefb7715e8764ccfe5d30c90d7444378b5f4f383ec95e2f1d7de@%3Cdev.spark.apache.org%3E>
>
> I believe we can handle them after 2.4.4 RC1 (or concurrently.)
>
> Hi, Kazuaki.
> Could you start a separate email thread for 2.3.4 release?
>
> Bests,
> Dongjoon.
>
>
> On Thu, Aug 15, 2019 at 8:43 AM Sean Owen <*sro...@gmail.com*
> > wrote:
> While we're on the topic:
>
> In theory, branch 2.3 is meant to be unsupported as of right about now.
>
> There are 69 fixes in branch 2.3 since 2.3.3 was released in Februrary:
> *https://issues.apache.org/jira/projects/SPARK/versions/12344844*
> <https://issues.apache.org/jira/projects/SPARK/versions/12344844>
>
> Some look moderately important.
>
> Should we also, or first, cut 2.3.4 to end the 2.3.x line?
>
> On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun <*dongjoon.h...@gmail.com*
> > wrote:
> >
> > Hi, All.
> >
> > Spark 2.4.3 was released three months ago (8th May).
> > As of today (13th August), there are 112 commits (75 JIRAs) in
> `branch-24` since 2.4.3.
> >
> > It would be great if we can have Spark 2.4.4.
> > Shall we start `2.4.4 RC1` next Monday (19th August)?
> >
> > Last time, there was a request for K8s issue and now I'm waiting for
> SPARK-27900.
> > Please let me know if there is another issue.
> >
> > Thanks,
> > Dongjoon.
>
>

Re: Release Apache Spark 2.4.4

[Kazuaki Ishizaki]

Sure, I will launch a separate e-mail thread for discussing 2.3.4 later.

Regards,
Kazuaki Ishizaki, Ph.D.



From:   Dongjoon Hyun 
To: Sean Owen , Kazuaki Ishizaki 
Cc: dev 
Date:   2019/08/16 05:10
Subject:[EXTERNAL] Re: Release Apache Spark 2.4.4



+1 for that.

Kazuaki volunteered for 2.3.4 release last month. AFAIK, he has been 
preparing that.

- 
https://lists.apache.org/thread.html/6fafeefb7715e8764ccfe5d30c90d7444378b5f4f383ec95e2f1d7de@%3Cdev.spark.apache.org%3E

I believe we can handle them after 2.4.4 RC1 (or concurrently.)

Hi, Kazuaki.
Could you start a separate email thread for 2.3.4 release?

Bests,
Dongjoon.


On Thu, Aug 15, 2019 at 8:43 AM Sean Owen  wrote:
While we're on the topic:

In theory, branch 2.3 is meant to be unsupported as of right about now.

There are 69 fixes in branch 2.3 since 2.3.3 was released in Februrary:
https://issues.apache.org/jira/projects/SPARK/versions/12344844

Some look moderately important.

Should we also, or first, cut 2.3.4 to end the 2.3.x line?

On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun  
wrote:
>
> Hi, All.
>
> Spark 2.4.3 was released three months ago (8th May).
> As of today (13th August), there are 112 commits (75 JIRAs) in 
`branch-24` since 2.4.3.
>
> It would be great if we can have Spark 2.4.4.
> Shall we start `2.4.4 RC1` next Monday (19th August)?
>
> Last time, there was a request for K8s issue and now I'm waiting for 
SPARK-27900.
> Please let me know if there is another issue.
>
> Thanks,
> Dongjoon.

Re: Release Apache Spark 2.4.4

[Dongjoon Hyun]

+1 for that.

Kazuaki volunteered for 2.3.4 release last month. AFAIK, he has been
preparing that.

-
https://lists.apache.org/thread.html/6fafeefb7715e8764ccfe5d30c90d7444378b5f4f383ec95e2f1d7de@%3Cdev.spark.apache.org%3E

I believe we can handle them after 2.4.4 RC1 (or concurrently.)

Hi, Kazuaki.
Could you start a separate email thread for 2.3.4 release?

Bests,
Dongjoon.


On Thu, Aug 15, 2019 at 8:43 AM Sean Owen  wrote:

> While we're on the topic:
>
> In theory, branch 2.3 is meant to be unsupported as of right about now.
>
> There are 69 fixes in branch 2.3 since 2.3.3 was released in Februrary:
> https://issues.apache.org/jira/projects/SPARK/versions/12344844
>
> Some look moderately important.
>
> Should we also, or first, cut 2.3.4 to end the 2.3.x line?
>
> On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun 
> wrote:
> >
> > Hi, All.
> >
> > Spark 2.4.3 was released three months ago (8th May).
> > As of today (13th August), there are 112 commits (75 JIRAs) in
> `branch-24` since 2.4.3.
> >
> > It would be great if we can have Spark 2.4.4.
> > Shall we start `2.4.4 RC1` next Monday (19th August)?
> >
> > Last time, there was a request for K8s issue and now I'm waiting for
> SPARK-27900.
> > Please let me know if there is another issue.
> >
> > Thanks,
> > Dongjoon.
>

Re: Release Apache Spark 2.4.4

[Sean Owen]

While we're on the topic:

In theory, branch 2.3 is meant to be unsupported as of right about now.

There are 69 fixes in branch 2.3 since 2.3.3 was released in Februrary:
https://issues.apache.org/jira/projects/SPARK/versions/12344844

Some look moderately important.

Should we also, or first, cut 2.3.4 to end the 2.3.x line?

On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun  wrote:
>
> Hi, All.
>
> Spark 2.4.3 was released three months ago (8th May).
> As of today (13th August), there are 112 commits (75 JIRAs) in `branch-24` 
> since 2.4.3.
>
> It would be great if we can have Spark 2.4.4.
> Shall we start `2.4.4 RC1` next Monday (19th August)?
>
> Last time, there was a request for K8s issue and now I'm waiting for 
> SPARK-27900.
> Please let me know if there is another issue.
>
> Thanks,
> Dongjoon.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Release Apache Spark 2.4.4

[Dongjoon Hyun]

Thank you, DB, Takeshi, Hyukjin, Sean, Kazuaki, Holden, Wenchen!
I'll create tag for 2.4.4-rc1 next Monday.

For SPARK-27234, it looks like that to me, too.

Thanks,
Dongjoon.


On Wed, Aug 14, 2019 at 9:13 AM Holden Karau  wrote:

> That looks like more of a feature than a bug fix unless I’m missing
> something?
>
> On Tue, Aug 13, 2019 at 11:58 PM Hyukjin Kwon  wrote:
>
>> Adding Shixiong
>>
>> WDYT?
>>
>> 2019년 8월 14일 (수) 오후 2:30, Terry Kim 님이 작성:
>>
>>> Can the following be included?
>>>
>>> [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch
>>> in EpochTracker (to support Python UDFs)
>>> 
>>>
>>> Thanks,
>>> Terry
>>>
>>> On Tue, Aug 13, 2019 at 10:24 PM Wenchen Fan 
>>> wrote:
>>>
 +1

 On Wed, Aug 14, 2019 at 12:52 PM Holden Karau 
 wrote:

> +1
> Does anyone have any critical fixes they’d like to see in 2.4.4?
>
> On Tue, Aug 13, 2019 at 5:22 PM Sean Owen  wrote:
>
>> Seems fine to me if there are enough valuable fixes to justify another
>> release. If there are any other important fixes imminent, it's fine to
>> wait for those.
>>
>>
>> On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun <
>> dongjoon.h...@gmail.com> wrote:
>> >
>> > Hi, All.
>> >
>> > Spark 2.4.3 was released three months ago (8th May).
>> > As of today (13th August), there are 112 commits (75 JIRAs) in
>> `branch-24` since 2.4.3.
>> >
>> > It would be great if we can have Spark 2.4.4.
>> > Shall we start `2.4.4 RC1` next Monday (19th August)?
>> >
>> > Last time, there was a request for K8s issue and now I'm waiting
>> for SPARK-27900.
>> > Please let me know if there is another issue.
>> >
>> > Thanks,
>> > Dongjoon.
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
 --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: Release Apache Spark 2.4.4

[Holden Karau]

That looks like more of a feature than a bug fix unless I’m missing
something?

On Tue, Aug 13, 2019 at 11:58 PM Hyukjin Kwon  wrote:

> Adding Shixiong
>
> WDYT?
>
> 2019년 8월 14일 (수) 오후 2:30, Terry Kim 님이 작성:
>
>> Can the following be included?
>>
>> [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in
>> EpochTracker (to support Python UDFs)
>> 
>>
>> Thanks,
>> Terry
>>
>> On Tue, Aug 13, 2019 at 10:24 PM Wenchen Fan  wrote:
>>
>>> +1
>>>
>>> On Wed, Aug 14, 2019 at 12:52 PM Holden Karau 
>>> wrote:
>>>
 +1
 Does anyone have any critical fixes they’d like to see in 2.4.4?

 On Tue, Aug 13, 2019 at 5:22 PM Sean Owen  wrote:

> Seems fine to me if there are enough valuable fixes to justify another
> release. If there are any other important fixes imminent, it's fine to
> wait for those.
>
>
> On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun 
> wrote:
> >
> > Hi, All.
> >
> > Spark 2.4.3 was released three months ago (8th May).
> > As of today (13th August), there are 112 commits (75 JIRAs) in
> `branch-24` since 2.4.3.
> >
> > It would be great if we can have Spark 2.4.4.
> > Shall we start `2.4.4 RC1` next Monday (19th August)?
> >
> > Last time, there was a request for K8s issue and now I'm waiting for
> SPARK-27900.
> > Please let me know if there is another issue.
> >
> > Thanks,
> > Dongjoon.
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
> --
 Twitter: https://twitter.com/holdenkarau
 Books (Learning Spark, High Performance Spark, etc.):
 https://amzn.to/2MaRAG9  
 YouTube Live Streams: https://www.youtube.com/user/holdenkarau

>>> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Release Apache Spark 2.4.4

[Hyukjin Kwon]

Adding Shixiong

WDYT?

2019년 8월 14일 (수) 오후 2:30, Terry Kim 님이 작성:

> Can the following be included?
>
> [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in
> EpochTracker (to support Python UDFs)
> 
>
> Thanks,
> Terry
>
> On Tue, Aug 13, 2019 at 10:24 PM Wenchen Fan  wrote:
>
>> +1
>>
>> On Wed, Aug 14, 2019 at 12:52 PM Holden Karau 
>> wrote:
>>
>>> +1
>>> Does anyone have any critical fixes they’d like to see in 2.4.4?
>>>
>>> On Tue, Aug 13, 2019 at 5:22 PM Sean Owen  wrote:
>>>
 Seems fine to me if there are enough valuable fixes to justify another
 release. If there are any other important fixes imminent, it's fine to
 wait for those.


 On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun 
 wrote:
 >
 > Hi, All.
 >
 > Spark 2.4.3 was released three months ago (8th May).
 > As of today (13th August), there are 112 commits (75 JIRAs) in
 `branch-24` since 2.4.3.
 >
 > It would be great if we can have Spark 2.4.4.
 > Shall we start `2.4.4 RC1` next Monday (19th August)?
 >
 > Last time, there was a request for K8s issue and now I'm waiting for
 SPARK-27900.
 > Please let me know if there is another issue.
 >
 > Thanks,
 > Dongjoon.

 -
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

 --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

Re: Release Apache Spark 2.4.4

[Terry Kim]

Can the following be included?

[SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in
EpochTracker (to support Python UDFs)


Thanks,
Terry

On Tue, Aug 13, 2019 at 10:24 PM Wenchen Fan  wrote:

> +1
>
> On Wed, Aug 14, 2019 at 12:52 PM Holden Karau 
> wrote:
>
>> +1
>> Does anyone have any critical fixes they’d like to see in 2.4.4?
>>
>> On Tue, Aug 13, 2019 at 5:22 PM Sean Owen  wrote:
>>
>>> Seems fine to me if there are enough valuable fixes to justify another
>>> release. If there are any other important fixes imminent, it's fine to
>>> wait for those.
>>>
>>>
>>> On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun 
>>> wrote:
>>> >
>>> > Hi, All.
>>> >
>>> > Spark 2.4.3 was released three months ago (8th May).
>>> > As of today (13th August), there are 112 commits (75 JIRAs) in
>>> `branch-24` since 2.4.3.
>>> >
>>> > It would be great if we can have Spark 2.4.4.
>>> > Shall we start `2.4.4 RC1` next Monday (19th August)?
>>> >
>>> > Last time, there was a request for K8s issue and now I'm waiting for
>>> SPARK-27900.
>>> > Please let me know if there is another issue.
>>> >
>>> > Thanks,
>>> > Dongjoon.
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Re: Release Apache Spark 2.4.4

[Wenchen Fan]

+1

On Wed, Aug 14, 2019 at 12:52 PM Holden Karau  wrote:

> +1
> Does anyone have any critical fixes they’d like to see in 2.4.4?
>
> On Tue, Aug 13, 2019 at 5:22 PM Sean Owen  wrote:
>
>> Seems fine to me if there are enough valuable fixes to justify another
>> release. If there are any other important fixes imminent, it's fine to
>> wait for those.
>>
>>
>> On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun 
>> wrote:
>> >
>> > Hi, All.
>> >
>> > Spark 2.4.3 was released three months ago (8th May).
>> > As of today (13th August), there are 112 commits (75 JIRAs) in
>> `branch-24` since 2.4.3.
>> >
>> > It would be great if we can have Spark 2.4.4.
>> > Shall we start `2.4.4 RC1` next Monday (19th August)?
>> >
>> > Last time, there was a request for K8s issue and now I'm waiting for
>> SPARK-27900.
>> > Please let me know if there is another issue.
>> >
>> > Thanks,
>> > Dongjoon.
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: Release Apache Spark 2.4.4

[Holden Karau]

+1
Does anyone have any critical fixes they’d like to see in 2.4.4?

On Tue, Aug 13, 2019 at 5:22 PM Sean Owen  wrote:

> Seems fine to me if there are enough valuable fixes to justify another
> release. If there are any other important fixes imminent, it's fine to
> wait for those.
>
>
> On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun 
> wrote:
> >
> > Hi, All.
> >
> > Spark 2.4.3 was released three months ago (8th May).
> > As of today (13th August), there are 112 commits (75 JIRAs) in
> `branch-24` since 2.4.3.
> >
> > It would be great if we can have Spark 2.4.4.
> > Shall we start `2.4.4 RC1` next Monday (19th August)?
> >
> > Last time, there was a request for K8s issue and now I'm waiting for
> SPARK-27900.
> > Please let me know if there is another issue.
> >
> > Thanks,
> > Dongjoon.
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

RE: Release Apache Spark 2.4.4

[Kazuaki Ishizaki]

Thanks, Dongjoon!
+1

Kazuaki Ishizaki,



From:   Hyukjin Kwon 
To: Takeshi Yamamuro 
Cc: Dongjoon Hyun , dev 
, User 
Date:   2019/08/14 09:21
Subject:[EXTERNAL] Re: Release Apache Spark 2.4.4



+1

2019년 8월 14일 (수) 오전 9:13, Takeshi Yamamuro 님
이 작성:
Hi,

Thanks for your notification, Dongjoon!
I put some links for the other committers/PMCs to access the info easily:

A commit list in github from the last release: 
https://github.com/apache/spark/compare/5ac2014e6c118fbeb1fe8e5c8064c4a8ee9d182a...branch-2.4
A issue list in jira: 
https://issues.apache.org/jira/projects/SPARK/versions/12345466#release-report-tab-body
The 5 correctness issues resolved in branch-2.4:
https://issues.apache.org/jira/browse/SPARK-27798?jql=project%20%3D%2012315420%20AND%20fixVersion%20%3D%2012345466%20AND%20labels%20in%20(%27correctness%27)%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC

Anyway, +1

Best,
Takeshi

On Wed, Aug 14, 2019 at 8:25 AM DB Tsai  wrote:
+1

On Tue, Aug 13, 2019 at 4:16 PM Dongjoon Hyun  
wrote:
>
> Hi, All.
>
> Spark 2.4.3 was released three months ago (8th May).
> As of today (13th August), there are 112 commits (75 JIRAs) in 
`branch-24` since 2.4.3.
>
> It would be great if we can have Spark 2.4.4.
> Shall we start `2.4.4 RC1` next Monday (19th August)?
>
> Last time, there was a request for K8s issue and now I'm waiting for 
SPARK-27900.
> Please let me know if there is another issue.
>
> Thanks,
> Dongjoon.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org



-- 
---
Takeshi Yamamuro

Re: Release Apache Spark 2.4.4

[Sean Owen]

Seems fine to me if there are enough valuable fixes to justify another
release. If there are any other important fixes imminent, it's fine to
wait for those.


On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun  wrote:
>
> Hi, All.
>
> Spark 2.4.3 was released three months ago (8th May).
> As of today (13th August), there are 112 commits (75 JIRAs) in `branch-24` 
> since 2.4.3.
>
> It would be great if we can have Spark 2.4.4.
> Shall we start `2.4.4 RC1` next Monday (19th August)?
>
> Last time, there was a request for K8s issue and now I'm waiting for 
> SPARK-27900.
> Please let me know if there is another issue.
>
> Thanks,
> Dongjoon.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Release Apache Spark 2.4.4

[Hyukjin Kwon]

+1

2019년 8월 14일 (수) 오전 9:13, Takeshi Yamamuro 님이 작성:

> Hi,
>
> Thanks for your notification, Dongjoon!
> I put some links for the other committers/PMCs to access the info easily:
>
> A commit list in github from the last release:
> https://github.com/apache/spark/compare/5ac2014e6c118fbeb1fe8e5c8064c4a8ee9d182a...branch-2.4
> A issue list in jira:
> https://issues.apache.org/jira/projects/SPARK/versions/12345466#release-report-tab-body
> The 5 correctness issues resolved in branch-2.4:
>
> https://issues.apache.org/jira/browse/SPARK-27798?jql=project%20%3D%2012315420%20AND%20fixVersion%20%3D%2012345466%20AND%20labels%20in%20(%27correctness%27)%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC
>
> Anyway, +1
>
> Best,
> Takeshi
>
> On Wed, Aug 14, 2019 at 8:25 AM DB Tsai  wrote:
>
>> +1
>>
>> On Tue, Aug 13, 2019 at 4:16 PM Dongjoon Hyun 
>> wrote:
>> >
>> > Hi, All.
>> >
>> > Spark 2.4.3 was released three months ago (8th May).
>> > As of today (13th August), there are 112 commits (75 JIRAs) in
>> `branch-24` since 2.4.3.
>> >
>> > It would be great if we can have Spark 2.4.4.
>> > Shall we start `2.4.4 RC1` next Monday (19th August)?
>> >
>> > Last time, there was a request for K8s issue and now I'm waiting for
>> SPARK-27900.
>> > Please let me know if there is another issue.
>> >
>> > Thanks,
>> > Dongjoon.
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>
> --
> ---
> Takeshi Yamamuro
>

Re: Release Apache Spark 2.4.4

[Takeshi Yamamuro]

Hi,

Thanks for your notification, Dongjoon!
I put some links for the other committers/PMCs to access the info easily:

A commit list in github from the last release:
https://github.com/apache/spark/compare/5ac2014e6c118fbeb1fe8e5c8064c4a8ee9d182a...branch-2.4
A issue list in jira:
https://issues.apache.org/jira/projects/SPARK/versions/12345466#release-report-tab-body
The 5 correctness issues resolved in branch-2.4:
https://issues.apache.org/jira/browse/SPARK-27798?jql=project%20%3D%2012315420%20AND%20fixVersion%20%3D%2012345466%20AND%20labels%20in%20(%27correctness%27)%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC

Anyway, +1

Best,
Takeshi

On Wed, Aug 14, 2019 at 8:25 AM DB Tsai  wrote:

> +1
>
> On Tue, Aug 13, 2019 at 4:16 PM Dongjoon Hyun 
> wrote:
> >
> > Hi, All.
> >
> > Spark 2.4.3 was released three months ago (8th May).
> > As of today (13th August), there are 112 commits (75 JIRAs) in
> `branch-24` since 2.4.3.
> >
> > It would be great if we can have Spark 2.4.4.
> > Shall we start `2.4.4 RC1` next Monday (19th August)?
> >
> > Last time, there was a request for K8s issue and now I'm waiting for
> SPARK-27900.
> > Please let me know if there is another issue.
> >
> > Thanks,
> > Dongjoon.
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

-- 
---
Takeshi Yamamuro

Re: Release Apache Spark 2.4.4

[DB Tsai]

+1

On Tue, Aug 13, 2019 at 4:16 PM Dongjoon Hyun  wrote:
>
> Hi, All.
>
> Spark 2.4.3 was released three months ago (8th May).
> As of today (13th August), there are 112 commits (75 JIRAs) in `branch-24` 
> since 2.4.3.
>
> It would be great if we can have Spark 2.4.4.
> Shall we start `2.4.4 RC1` next Monday (19th August)?
>
> Last time, there was a request for K8s issue and now I'm waiting for 
> SPARK-27900.
> Please let me know if there is another issue.
>
> Thanks,
> Dongjoon.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Release Apache Spark 2.4.4

[Dongjoon Hyun]

Hi, All.

Spark 2.4.3 was released three months ago (8th May).
As of today (13th August), there are 112 commits (75 JIRAs) in `branch-24`
since 2.4.3.

It would be great if we can have Spark 2.4.4.
Shall we start `2.4.4 RC1` next Monday (19th August)?

Last time, there was a request for K8s issue and now I'm waiting for
SPARK-27900.
Please let me know if there is another issue.

Thanks,
Dongjoon.

Re: Re: Release Apache Spark 2.4.4 before 3.0.0

[Dongjoon Hyun]

Thank you for volunteering for 2.3.4 release manager, Kazuaki!
It's great to see a new release manager in advance. :D

Thank you for reply, Stavros.
In addition to that issue, I'm also monitoring some other K8s issues and
PRs.
But, I'm not sure we can have that because some PRs seems to fail at
building consensus (even for 3.0.0).
In any way, could you ping the reviewers once more on those PRs which you
have concerns?
If it is merged into `branch-2.4`, it will be Apache Spark 2.4.4 of course.

Bests,
Dongjoon.


On Tue, Jul 16, 2019 at 4:00 AM Kazuaki Ishizaki 
wrote:

> Thank you Dongjoon for being a release manager.
>
> If the assumed dates are ok, I would like to volunteer for an 2.3.4
> release manager.
>
> Best Regards,
> Kazuaki Ishizaki,
>
>
>
> From:Dongjoon Hyun 
> To:dev , "user @spark" <
> u...@spark.apache.org>, Apache Spark PMC 
> Date:    2019/07/13 07:18
> Subject:[EXTERNAL] Re: Release Apache Spark 2.4.4 before 3.0.0
> --
>
>
>
> Thank you, Jacek.
>
> BTW, I added `@private` since we need PMC's help to make an Apache Spark
> release.
>
> Can I get more feedbacks from the other PMC members?
>
> Please me know if you have any concerns (e.g. Release date or Release
> manager?)
>
> As one of the community members, I assumed the followings (if we are on
> schedule).
>
> - 2.4.4 at the end of July
> - 2.3.4 at the end of August (since 2.3.0 was released at the end of
> February 2018)
> - 3.0.0 (possibily September?)
> - 3.1.0 (January 2020?)
>
> Bests,
> Dongjoon.
>
>
> On Thu, Jul 11, 2019 at 1:30 PM Jacek Laskowski <*ja...@japila.pl*
> > wrote:
> Hi,
>
> Thanks Dongjoon Hyun for stepping up as a release manager!
> Much appreciated.
>
> If there's a volunteer to cut a release, I'm always to support it.
>
> In addition, the more frequent releases the better for end users so they
> have a choice to upgrade and have all the latest fixes or wait. It's their
> call not ours (when we'd keep them waiting).
>
> My big 2 yes'es for the release!
>
> Jacek
>
>
> On Tue, 9 Jul 2019, 18:15 Dongjoon Hyun, <*dongjoon.h...@gmail.com*
> > wrote:
> Hi, All.
>
> Spark 2.4.3 was released two months ago (8th May).
>
> As of today (9th July), there exist 45 fixes in `branch-2.4` including the
> following correctness or blocker issues.
>
> - SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for
> decimals not fitting in long
> - SPARK-26045 Error in the spark 2.4 release package with the
> spark-avro_2.11 dependency
> - SPARK-27798 from_avro can modify variables in other rows in local
> mode
> - SPARK-27907 HiveUDAF should return NULL in case of 0 rows
> - SPARK-28157 Make SHS clear KVStore LogInfo for the blacklist entries
> - SPARK-28308 CalendarInterval sub-second part should be padded before
> parsing
>
> It would be great if we can have Spark 2.4.4 before we are going to get
> busier for 3.0.0.
> If it's okay, I'd like to volunteer for an 2.4.4 release manager to roll
> it next Monday. (15th July).
> How do you think about this?
>
> Bests,
> Dongjoon.
>
>

Re: Re: Release Apache Spark 2.4.4 before 3.0.0

[Kazuaki Ishizaki]

Thank you Dongjoon for being a release manager.

If the assumed dates are ok, I would like to volunteer for an 2.3.4 
release manager.

Best Regards,
Kazuaki Ishizaki,



From:   Dongjoon Hyun 
To: dev , "user @spark" , 
Apache Spark PMC 
Date:   2019/07/13 07:18
Subject:[EXTERNAL] Re: Release Apache Spark 2.4.4 before 3.0.0



Thank you, Jacek.

BTW, I added `@private` since we need PMC's help to make an Apache Spark 
release.

Can I get more feedbacks from the other PMC members?

Please me know if you have any concerns (e.g. Release date or Release 
manager?)

As one of the community members, I assumed the followings (if we are on 
schedule).

- 2.4.4 at the end of July
- 2.3.4 at the end of August (since 2.3.0 was released at the end of 
February 2018)
- 3.0.0 (possibily September?)
- 3.1.0 (January 2020?)

Bests,
Dongjoon.


On Thu, Jul 11, 2019 at 1:30 PM Jacek Laskowski  wrote:
Hi,

Thanks Dongjoon Hyun for stepping up as a release manager! 
Much appreciated. 

If there's a volunteer to cut a release, I'm always to support it.

In addition, the more frequent releases the better for end users so they 
have a choice to upgrade and have all the latest fixes or wait. It's their 
call not ours (when we'd keep them waiting).

My big 2 yes'es for the release!

Jacek


On Tue, 9 Jul 2019, 18:15 Dongjoon Hyun,  wrote:
Hi, All.

Spark 2.4.3 was released two months ago (8th May).

As of today (9th July), there exist 45 fixes in `branch-2.4` including the 
following correctness or blocker issues.

- SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for 
decimals not fitting in long
- SPARK-26045 Error in the spark 2.4 release package with the 
spark-avro_2.11 dependency
- SPARK-27798 from_avro can modify variables in other rows in local 
mode
- SPARK-27907 HiveUDAF should return NULL in case of 0 rows
- SPARK-28157 Make SHS clear KVStore LogInfo for the blacklist entries
- SPARK-28308 CalendarInterval sub-second part should be padded before 
parsing

It would be great if we can have Spark 2.4.4 before we are going to get 
busier for 3.0.0.
If it's okay, I'd like to volunteer for an 2.4.4 release manager to roll 
it next Monday. (15th July).
How do you think about this?

Bests,
Dongjoon.

Re: Release Apache Spark 2.4.4 before 3.0.0

[Stavros Kontopoulos]

Hi Dongjoon,

Should we also consider fixing
https://issues.apache.org/jira/browse/SPARK-27812 before the cut?

Best,
Stavros

On Mon, Jul 15, 2019 at 7:04 PM Dongjoon Hyun 
wrote:

> Hi, Apache Spark PMC members.
>
> Can we cut Apache Spark 2.4.4 next Monday (22nd July)?
>
> Bests,
> Dongjoon.
>
>
> On Fri, Jul 12, 2019 at 3:18 PM Dongjoon Hyun 
> wrote:
>
>> Thank you, Jacek.
>>
>> BTW, I added `@private` since we need PMC's help to make an Apache Spark
>> release.
>>
>> Can I get more feedbacks from the other PMC members?
>>
>> Please me know if you have any concerns (e.g. Release date or Release
>> manager?)
>>
>> As one of the community members, I assumed the followings (if we are on
>> schedule).
>>
>> - 2.4.4 at the end of July
>> - 2.3.4 at the end of August (since 2.3.0 was released at the end of
>> February 2018)
>> - 3.0.0 (possibily September?)
>> - 3.1.0 (January 2020?)
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Thu, Jul 11, 2019 at 1:30 PM Jacek Laskowski  wrote:
>>
>>> Hi,
>>>
>>> Thanks Dongjoon Hyun for stepping up as a release manager!
>>> Much appreciated.
>>>
>>> If there's a volunteer to cut a release, I'm always to support it.
>>>
>>> In addition, the more frequent releases the better for end users so they
>>> have a choice to upgrade and have all the latest fixes or wait. It's their
>>> call not ours (when we'd keep them waiting).
>>>
>>> My big 2 yes'es for the release!
>>>
>>> Jacek
>>>
>>>
>>> On Tue, 9 Jul 2019, 18:15 Dongjoon Hyun, 
>>> wrote:
>>>
 Hi, All.

 Spark 2.4.3 was released two months ago (8th May).

 As of today (9th July), there exist 45 fixes in `branch-2.4` including
 the following correctness or blocker issues.

 - SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for
 decimals not fitting in long
 - SPARK-26045 Error in the spark 2.4 release package with the
 spark-avro_2.11 dependency
 - SPARK-27798 from_avro can modify variables in other rows in local
 mode
 - SPARK-27907 HiveUDAF should return NULL in case of 0 rows
 - SPARK-28157 Make SHS clear KVStore LogInfo for the blacklist
 entries
 - SPARK-28308 CalendarInterval sub-second part should be padded
 before parsing

 It would be great if we can have Spark 2.4.4 before we are going to get
 busier for 3.0.0.
 If it's okay, I'd like to volunteer for an 2.4.4 release manager to
 roll it next Monday. (15th July).
 How do you think about this?

 Bests,
 Dongjoon.

>>>

Re: Release Apache Spark 2.4.4 before 3.0.0

[Dongjoon Hyun]

Hi, Apache Spark PMC members.

Can we cut Apache Spark 2.4.4 next Monday (22nd July)?

Bests,
Dongjoon.


On Fri, Jul 12, 2019 at 3:18 PM Dongjoon Hyun 
wrote:

> Thank you, Jacek.
>
> BTW, I added `@private` since we need PMC's help to make an Apache Spark
> release.
>
> Can I get more feedbacks from the other PMC members?
>
> Please me know if you have any concerns (e.g. Release date or Release
> manager?)
>
> As one of the community members, I assumed the followings (if we are on
> schedule).
>
> - 2.4.4 at the end of July
> - 2.3.4 at the end of August (since 2.3.0 was released at the end of
> February 2018)
> - 3.0.0 (possibily September?)
> - 3.1.0 (January 2020?)
>
> Bests,
> Dongjoon.
>
>
> On Thu, Jul 11, 2019 at 1:30 PM Jacek Laskowski  wrote:
>
>> Hi,
>>
>> Thanks Dongjoon Hyun for stepping up as a release manager!
>> Much appreciated.
>>
>> If there's a volunteer to cut a release, I'm always to support it.
>>
>> In addition, the more frequent releases the better for end users so they
>> have a choice to upgrade and have all the latest fixes or wait. It's their
>> call not ours (when we'd keep them waiting).
>>
>> My big 2 yes'es for the release!
>>
>> Jacek
>>
>>
>> On Tue, 9 Jul 2019, 18:15 Dongjoon Hyun,  wrote:
>>
>>> Hi, All.
>>>
>>> Spark 2.4.3 was released two months ago (8th May).
>>>
>>> As of today (9th July), there exist 45 fixes in `branch-2.4` including
>>> the following correctness or blocker issues.
>>>
>>> - SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for
>>> decimals not fitting in long
>>> - SPARK-26045 Error in the spark 2.4 release package with the
>>> spark-avro_2.11 dependency
>>> - SPARK-27798 from_avro can modify variables in other rows in local
>>> mode
>>> - SPARK-27907 HiveUDAF should return NULL in case of 0 rows
>>> - SPARK-28157 Make SHS clear KVStore LogInfo for the blacklist
>>> entries
>>> - SPARK-28308 CalendarInterval sub-second part should be padded
>>> before parsing
>>>
>>> It would be great if we can have Spark 2.4.4 before we are going to get
>>> busier for 3.0.0.
>>> If it's okay, I'd like to volunteer for an 2.4.4 release manager to roll
>>> it next Monday. (15th July).
>>> How do you think about this?
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>

Re: Release Apache Spark 2.4.4 before 3.0.0

[Dongjoon Hyun]

Thank you, Jacek.

BTW, I added `@private` since we need PMC's help to make an Apache Spark
release.

Can I get more feedbacks from the other PMC members?

Please me know if you have any concerns (e.g. Release date or Release
manager?)

As one of the community members, I assumed the followings (if we are on
schedule).

- 2.4.4 at the end of July
- 2.3.4 at the end of August (since 2.3.0 was released at the end of
February 2018)
- 3.0.0 (possibily September?)
- 3.1.0 (January 2020?)

Bests,
Dongjoon.


On Thu, Jul 11, 2019 at 1:30 PM Jacek Laskowski  wrote:

> Hi,
>
> Thanks Dongjoon Hyun for stepping up as a release manager!
> Much appreciated.
>
> If there's a volunteer to cut a release, I'm always to support it.
>
> In addition, the more frequent releases the better for end users so they
> have a choice to upgrade and have all the latest fixes or wait. It's their
> call not ours (when we'd keep them waiting).
>
> My big 2 yes'es for the release!
>
> Jacek
>
>
> On Tue, 9 Jul 2019, 18:15 Dongjoon Hyun,  wrote:
>
>> Hi, All.
>>
>> Spark 2.4.3 was released two months ago (8th May).
>>
>> As of today (9th July), there exist 45 fixes in `branch-2.4` including
>> the following correctness or blocker issues.
>>
>> - SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for
>> decimals not fitting in long
>> - SPARK-26045 Error in the spark 2.4 release package with the
>> spark-avro_2.11 dependency
>> - SPARK-27798 from_avro can modify variables in other rows in local
>> mode
>> - SPARK-27907 HiveUDAF should return NULL in case of 0 rows
>> - SPARK-28157 Make SHS clear KVStore LogInfo for the blacklist entries
>> - SPARK-28308 CalendarInterval sub-second part should be padded
>> before parsing
>>
>> It would be great if we can have Spark 2.4.4 before we are going to get
>> busier for 3.0.0.
>> If it's okay, I'd like to volunteer for an 2.4.4 release manager to roll
>> it next Monday. (15th July).
>> How do you think about this?
>>
>> Bests,
>> Dongjoon.
>>
>

Re: Release Apache Spark 2.4.4 before 3.0.0

[Jacek Laskowski]

Hi,

Thanks Dongjoon Hyun for stepping up as a release manager!
Much appreciated.

If there's a volunteer to cut a release, I'm always to support it.

In addition, the more frequent releases the better for end users so they
have a choice to upgrade and have all the latest fixes or wait. It's their
call not ours (when we'd keep them waiting).

My big 2 yes'es for the release!

Jacek


On Tue, 9 Jul 2019, 18:15 Dongjoon Hyun,  wrote:

> Hi, All.
>
> Spark 2.4.3 was released two months ago (8th May).
>
> As of today (9th July), there exist 45 fixes in `branch-2.4` including the
> following correctness or blocker issues.
>
> - SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for
> decimals not fitting in long
> - SPARK-26045 Error in the spark 2.4 release package with the
> spark-avro_2.11 dependency
> - SPARK-27798 from_avro can modify variables in other rows in local
> mode
> - SPARK-27907 HiveUDAF should return NULL in case of 0 rows
> - SPARK-28157 Make SHS clear KVStore LogInfo for the blacklist entries
> - SPARK-28308 CalendarInterval sub-second part should be padded before
> parsing
>
> It would be great if we can have Spark 2.4.4 before we are going to get
> busier for 3.0.0.
> If it's okay, I'd like to volunteer for an 2.4.4 release manager to roll
> it next Monday. (15th July).
> How do you think about this?
>
> Bests,
> Dongjoon.
>

Re: Release Apache Spark 2.4.4 before 3.0.0

[Dongjoon Hyun]

Additionally, one more correctness patch landed yesterday.

- SPARK-28015 Check stringToDate() consumes entire input for the 
and -[m]m formats

Bests,
Dongjoon.


On Tue, Jul 9, 2019 at 10:11 AM Dongjoon Hyun 
wrote:

> Thank you for the reply, Sean. Sure. 2.4.x should be a LTS version.
>
> The main reason of 2.4.4 release (before 3.0.0) is to have a better basis
> for comparison to 3.0.0.
> For example, SPARK-27798 had an old bug, but its correctness issue is only
> exposed at Spark 2.4.3.
> It would be great if we can have a better basis.
>
> Bests,
> Dongjoon.
>
>
> On Tue, Jul 9, 2019 at 9:52 AM Sean Owen  wrote:
>
>> We will certainly want a 2.4.4 release eventually. In fact I'd expect
>> 2.4.x gets maintained for longer than the usual 18 months, as it's the
>> last 2.x branch.
>> It doesn't need to happen before 3.0, but could. Usually maintenance
>> releases happen 3-4 months apart and the last one was 2 months ago. If
>> these are significant issues, sure. It'll probably be August before
>> it's out anyway.
>>
>> On Tue, Jul 9, 2019 at 11:15 AM Dongjoon Hyun 
>> wrote:
>> >
>> > Hi, All.
>> >
>> > Spark 2.4.3 was released two months ago (8th May).
>> >
>> > As of today (9th July), there exist 45 fixes in `branch-2.4` including
>> the following correctness or blocker issues.
>> >
>> > - SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for
>> decimals not fitting in long
>> > - SPARK-26045 Error in the spark 2.4 release package with the
>> spark-avro_2.11 dependency
>> > - SPARK-27798 from_avro can modify variables in other rows in local
>> mode
>> > - SPARK-27907 HiveUDAF should return NULL in case of 0 rows
>> > - SPARK-28157 Make SHS clear KVStore LogInfo for the blacklist
>> entries
>> > - SPARK-28308 CalendarInterval sub-second part should be padded
>> before parsing
>> >
>> > It would be great if we can have Spark 2.4.4 before we are going to get
>> busier for 3.0.0.
>> > If it's okay, I'd like to volunteer for an 2.4.4 release manager to
>> roll it next Monday. (15th July).
>> > How do you think about this?
>> >
>> > Bests,
>> > Dongjoon.
>>
>

Re: Release Apache Spark 2.4.4 before 3.0.0

[Dongjoon Hyun]

Thank you for the reply, Sean. Sure. 2.4.x should be a LTS version.

The main reason of 2.4.4 release (before 3.0.0) is to have a better basis
for comparison to 3.0.0.
For example, SPARK-27798 had an old bug, but its correctness issue is only
exposed at Spark 2.4.3.
It would be great if we can have a better basis.

Bests,
Dongjoon.


On Tue, Jul 9, 2019 at 9:52 AM Sean Owen  wrote:

> We will certainly want a 2.4.4 release eventually. In fact I'd expect
> 2.4.x gets maintained for longer than the usual 18 months, as it's the
> last 2.x branch.
> It doesn't need to happen before 3.0, but could. Usually maintenance
> releases happen 3-4 months apart and the last one was 2 months ago. If
> these are significant issues, sure. It'll probably be August before
> it's out anyway.
>
> On Tue, Jul 9, 2019 at 11:15 AM Dongjoon Hyun 
> wrote:
> >
> > Hi, All.
> >
> > Spark 2.4.3 was released two months ago (8th May).
> >
> > As of today (9th July), there exist 45 fixes in `branch-2.4` including
> the following correctness or blocker issues.
> >
> > - SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for
> decimals not fitting in long
> > - SPARK-26045 Error in the spark 2.4 release package with the
> spark-avro_2.11 dependency
> > - SPARK-27798 from_avro can modify variables in other rows in local
> mode
> > - SPARK-27907 HiveUDAF should return NULL in case of 0 rows
> > - SPARK-28157 Make SHS clear KVStore LogInfo for the blacklist
> entries
> > - SPARK-28308 CalendarInterval sub-second part should be padded
> before parsing
> >
> > It would be great if we can have Spark 2.4.4 before we are going to get
> busier for 3.0.0.
> > If it's okay, I'd like to volunteer for an 2.4.4 release manager to roll
> it next Monday. (15th July).
> > How do you think about this?
> >
> > Bests,
> > Dongjoon.
>

Re: Release Apache Spark 2.4.4 before 3.0.0

[Sean Owen]

We will certainly want a 2.4.4 release eventually. In fact I'd expect
2.4.x gets maintained for longer than the usual 18 months, as it's the
last 2.x branch.
It doesn't need to happen before 3.0, but could. Usually maintenance
releases happen 3-4 months apart and the last one was 2 months ago. If
these are significant issues, sure. It'll probably be August before
it's out anyway.

On Tue, Jul 9, 2019 at 11:15 AM Dongjoon Hyun  wrote:
>
> Hi, All.
>
> Spark 2.4.3 was released two months ago (8th May).
>
> As of today (9th July), there exist 45 fixes in `branch-2.4` including the 
> following correctness or blocker issues.
>
> - SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for 
> decimals not fitting in long
> - SPARK-26045 Error in the spark 2.4 release package with the 
> spark-avro_2.11 dependency
> - SPARK-27798 from_avro can modify variables in other rows in local mode
> - SPARK-27907 HiveUDAF should return NULL in case of 0 rows
> - SPARK-28157 Make SHS clear KVStore LogInfo for the blacklist entries
> - SPARK-28308 CalendarInterval sub-second part should be padded before 
> parsing
>
> It would be great if we can have Spark 2.4.4 before we are going to get 
> busier for 3.0.0.
> If it's okay, I'd like to volunteer for an 2.4.4 release manager to roll it 
> next Monday. (15th July).
> How do you think about this?
>
> Bests,
> Dongjoon.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Release Apache Spark 2.4.4 before 3.0.0

[Dongjoon Hyun]

Hi, All.

Spark 2.4.3 was released two months ago (8th May).

As of today (9th July), there exist 45 fixes in `branch-2.4` including the
following correctness or blocker issues.

- SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for
decimals not fitting in long
- SPARK-26045 Error in the spark 2.4 release package with the
spark-avro_2.11 dependency
- SPARK-27798 from_avro can modify variables in other rows in local mode
- SPARK-27907 HiveUDAF should return NULL in case of 0 rows
- SPARK-28157 Make SHS clear KVStore LogInfo for the blacklist entries
- SPARK-28308 CalendarInterval sub-second part should be padded before
parsing

It would be great if we can have Spark 2.4.4 before we are going to get
busier for 3.0.0.
If it's okay, I'd like to volunteer for an 2.4.4 release manager to roll it
next Monday. (15th July).
How do you think about this?

Bests,
Dongjoon.