Re: [VOTE] SPARK 2.4.0 (RC3)

[Dongjoon Hyun]

Thank you, Wenchen. I made the minor document-only change PR.

https://github.com/apache/spark/pull/22781

Bests,
Dongjoon.



On Fri, Oct 19, 2018 at 6:07 PM Wenchen Fan  wrote:

> AFAIK we haven't tested Java 9+ yet, so I'm ok to change it.
>
> Hi Dongjoon can you make a PR for it? We can merge it very soon if we
> decide to do it.
>
> Thanks,
> Wenchen
>
> On Sat, Oct 20, 2018 at 5:27 AM Dongjoon Hyun 
> wrote:
>
>> From the document, should we be more specific with 'Java 8' instead of
>> 'Java 8+' because we don't build (or test) in the community with Java 9 ~
>> 11.
>>
>>
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/building-spark.html
>> > Building Spark using Maven requires Maven 3.3.9 or newer and Java 8+
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan  wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 2.4.0.
>>>
>>> The vote is open until October 1 PST and passes if a majority +1 PMC
>>> votes are cast, with
>>> a minimum of 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.4.0
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v2.4.0-rc3 (commit
>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>> https://github.com/apache/spark/tree/v2.4.0-rc3
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>>
>>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>>> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>>
>>> FAQ
>>>
>>> =
>>> How can I help test this release?
>>> =
>>>
>>> If you are a Spark user, you can help us test this release by taking
>>> an existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>> the current RC and see if anything important breaks, in the Java/Scala
>>> you can add the staging repository to your projects resolvers and test
>>> with the RC (make sure to clean up the artifact cache before/after so
>>> you don't end up building with a out of date RC going forward).
>>>
>>> ===
>>> What should happen to JIRA tickets still targeting 2.4.0?
>>> ===
>>>
>>> The current list of open tickets targeted at 2.4.0 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.0
>>>
>>> Committers should look at those and triage. Extremely important bug
>>> fixes, documentation, and API tweaks that impact compatibility should
>>> be worked on immediately. Everything else please retarget to an
>>> appropriate release.
>>>
>>> ==
>>> But my bug isn't fixed?
>>> ==
>>>
>>> In order to make timely releases, we will typically not hold the
>>> release unless the bug in question is a regression from the previous
>>> release. That being said, if there is something which is a regression
>>> that has not been correctly targeted please ping me or a committer to
>>> help target the issue.
>>>
>>

Re: [VOTE] SPARK 2.4.0 (RC3)

[Wenchen Fan]

AFAIK we haven't tested Java 9+ yet, so I'm ok to change it.

Hi Dongjoon can you make a PR for it? We can merge it very soon if we
decide to do it.

Thanks,
Wenchen

On Sat, Oct 20, 2018 at 5:27 AM Dongjoon Hyun 
wrote:

> From the document, should we be more specific with 'Java 8' instead of
> 'Java 8+' because we don't build (or test) in the community with Java 9 ~
> 11.
>
>
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/building-spark.html
> > Building Spark using Maven requires Maven 3.3.9 or newer and Java 8+
>
> Bests,
> Dongjoon.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan  wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 2.4.0.
>>
>> The vote is open until October 1 PST and passes if a majority +1 PMC
>> votes are cast, with
>> a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 2.4.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v2.4.0-rc3 (commit
>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>> https://github.com/apache/spark/tree/v2.4.0-rc3
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>
>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>
>> FAQ
>>
>> =
>> How can I help test this release?
>> =
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===
>> What should happen to JIRA tickets still targeting 2.4.0?
>> ===
>>
>> The current list of open tickets targeted at 2.4.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==
>> But my bug isn't fixed?
>> ==
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>

Re: [VOTE] SPARK 2.4.0 (RC3)

[Dongjoon Hyun]

>From the document, should we be more specific with 'Java 8' instead of
'Java 8+' because we don't build (or test) in the community with Java 9 ~
11.

https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/building-spark.html
> Building Spark using Maven requires Maven 3.3.9 or newer and Java 8+

Bests,
Dongjoon.


On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.0?
> ===
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>

Re: [VOTE] SPARK 2.4.0 (RC3)

[Koert Kuipers]

i deployed 2.4.0 RC3 on our dev cluster and ran into issue with spark shell
and jline. there is probably a simple workaround so this is not a serious
issue but just wanted to let you know.

https://issues.apache.org/jira/browse/SPARK-25783

On Mon, Oct 15, 2018 at 4:59 PM Imran Rashid 
wrote:

> I just discovered https://issues.apache.org/jira/browse/SPARK-25738 with
> some more testing.  I only marked it as critical, but seems pretty bad --
> I'll defer to others opinion
>
> On Sat, Oct 13, 2018 at 4:15 PM Dongjoon Hyun 
> wrote:
>
>> Yes. From my side, it's -1 for RC3.
>>
>> Bests,
>> Dongjoon.
>>
>> On Sat, Oct 13, 2018 at 1:24 PM Holden Karau 
>> wrote:
>>
>>> So if it's a blocker would you think this should be a -1?
>>>
>>> On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun 
>>> wrote:
>>>
 Hi, Holden.

 Since that's a performance at 2.4.0, I marked as `Blocker` four days
 ago.

 Bests,
 Dongjoon.


 On Fri, Oct 12, 2018 at 11:45 AM Holden Karau 
 wrote:

> Following up I just wanted to make sure this new blocker that Dongjoon
> designated is surfaced -
> https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker
>
> On Thu, Oct 11, 2018 at 2:05 PM Xiao Li  wrote:
>
>> -1. We have two correctness bugs:
>> https://issues.apache.org/jira/browse/SPARK-25714 and
>> https://issues.apache.org/jira/browse/SPARK-25708.
>>
>> Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
>>
>> Xiao
>>
>>
>> Sean Owen  于2018年10月11日周四 上午9:04写道：
>>
>>> This is a legitimate question about the behavior of ScalaUDF after
>>> the
>>> change to support 2.12:
>>> https://github.com/apache/spark/pull/22259#discussion_r224295469
>>> Not quite a blocker I think, but a potential gotcha we definitely
>>> need
>>> to highlight in release notes. There may be an argument for changing
>>> ScalaUDF again before the release. Have a look, anyone familiar with
>>> catalyst.
>>> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen  wrote:
>>> >
>>> > +1. I tested the source build against Scala 2.12 and common build
>>> > profiles. License and sigs look OK.
>>> >
>>> > No blockers; one critical:
>>> >
>>> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>>> >
>>> > I think this one is "won't fix" though? not trying to restore the
>>> behavior?
>>> >
>>> > Other items open for 2.4.0:
>>> >
>>> > SPARK-25347 Document image data source in doc site
>>> > SPARK-25584 Document libsvm data source in doc site
>>> > SPARK-25179 Document the features that require Pyarrow 0.10
>>> > SPARK-25507 Update documents for the new features in 2.4 release
>>> > SPARK-25346 Document Spark builtin data sources
>>> > SPARK-24464 Unit tests for MLlib's Instrumentation
>>> > SPARK-23197 Flaky test:
>>> spark.streaming.ReceiverSuite."receiver_life_cycle"
>>> > SPARK-22809 pyspark is sensitive to imports with dots
>>> > SPARK-21030 extend hint syntax to support any expression for
>>> Python and R
>>> >
>>> > Anyone know enough to close or retarget them? they don't look
>>> critical
>>> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document
>>> the
>>> > features that require Pyarrow 0.10" however sounds like it could
>>> have
>>> > been important for 2.4? if not a blocker.
>>> >
>>> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
>>> > there is some ongoing discussion there.
>>> >
>>> > I am evaluating
>>> > https://github.com/apache/spark/pull/22259#discussion_r224252642
>>> right
>>> > now.
>>> >
>>> >
>>> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan 
>>> wrote:
>>> > >
>>> > > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.0.
>>> > >
>>> > > The vote is open until October 1 PST and passes if a majority +1
>>> PMC votes are cast, with
>>> > > a minimum of 3 +1 votes.
>>> > >
>>> > > [ ] +1 Release this package as Apache Spark 2.4.0
>>> > > [ ] -1 Do not release this package because ...
>>> > >
>>> > > To learn more about Apache Spark, please see
>>> http://spark.apache.org/
>>> > >
>>> > > The tag to be voted on is v2.4.0-rc3 (commit
>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>> > > https://github.com/apache/spark/tree/v2.4.0-rc3
>>> > >
>>> > > The release files, including signatures, digests, etc. can be
>>> found at:
>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>> > >
>>> > > Signatures used for Spark RCs can be found 

Re: [VOTE] SPARK 2.4.0 (RC3)

[Imran Rashid]

I just discovered https://issues.apache.org/jira/browse/SPARK-25738 with
some more testing.  I only marked it as critical, but seems pretty bad --
I'll defer to others opinion

On Sat, Oct 13, 2018 at 4:15 PM Dongjoon Hyun 
wrote:

> Yes. From my side, it's -1 for RC3.
>
> Bests,
> Dongjoon.
>
> On Sat, Oct 13, 2018 at 1:24 PM Holden Karau  wrote:
>
>> So if it's a blocker would you think this should be a -1?
>>
>> On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun 
>> wrote:
>>
>>> Hi, Holden.
>>>
>>> Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Fri, Oct 12, 2018 at 11:45 AM Holden Karau 
>>> wrote:
>>>
 Following up I just wanted to make sure this new blocker that Dongjoon
 designated is surfaced -
 https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker

 On Thu, Oct 11, 2018 at 2:05 PM Xiao Li  wrote:

> -1. We have two correctness bugs:
> https://issues.apache.org/jira/browse/SPARK-25714 and
> https://issues.apache.org/jira/browse/SPARK-25708.
>
> Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
>
> Xiao
>
>
> Sean Owen  于2018年10月11日周四 上午9:04写道：
>
>> This is a legitimate question about the behavior of ScalaUDF after the
>> change to support 2.12:
>> https://github.com/apache/spark/pull/22259#discussion_r224295469
>> Not quite a blocker I think, but a potential gotcha we definitely need
>> to highlight in release notes. There may be an argument for changing
>> ScalaUDF again before the release. Have a look, anyone familiar with
>> catalyst.
>> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen  wrote:
>> >
>> > +1. I tested the source build against Scala 2.12 and common build
>> > profiles. License and sigs look OK.
>> >
>> > No blockers; one critical:
>> >
>> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>> >
>> > I think this one is "won't fix" though? not trying to restore the
>> behavior?
>> >
>> > Other items open for 2.4.0:
>> >
>> > SPARK-25347 Document image data source in doc site
>> > SPARK-25584 Document libsvm data source in doc site
>> > SPARK-25179 Document the features that require Pyarrow 0.10
>> > SPARK-25507 Update documents for the new features in 2.4 release
>> > SPARK-25346 Document Spark builtin data sources
>> > SPARK-24464 Unit tests for MLlib's Instrumentation
>> > SPARK-23197 Flaky test:
>> spark.streaming.ReceiverSuite."receiver_life_cycle"
>> > SPARK-22809 pyspark is sensitive to imports with dots
>> > SPARK-21030 extend hint syntax to support any expression for Python
>> and R
>> >
>> > Anyone know enough to close or retarget them? they don't look
>> critical
>> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document
>> the
>> > features that require Pyarrow 0.10" however sounds like it could
>> have
>> > been important for 2.4? if not a blocker.
>> >
>> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
>> > there is some ongoing discussion there.
>> >
>> > I am evaluating
>> > https://github.com/apache/spark/pull/22259#discussion_r224252642
>> right
>> > now.
>> >
>> >
>> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan 
>> wrote:
>> > >
>> > > Please vote on releasing the following candidate as Apache Spark
>> version 2.4.0.
>> > >
>> > > The vote is open until October 1 PST and passes if a majority +1
>> PMC votes are cast, with
>> > > a minimum of 3 +1 votes.
>> > >
>> > > [ ] +1 Release this package as Apache Spark 2.4.0
>> > > [ ] -1 Do not release this package because ...
>> > >
>> > > To learn more about Apache Spark, please see
>> http://spark.apache.org/
>> > >
>> > > The tag to be voted on is v2.4.0-rc3 (commit
>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>> > > https://github.com/apache/spark/tree/v2.4.0-rc3
>> > >
>> > > The release files, including signatures, digests, etc. can be
>> found at:
>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>> > >
>> > > Signatures used for Spark RCs can be found in this file:
>> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
>> > >
>> > > The staging repository for this release can be found at:
>> > >
>> https://repository.apache.org/content/repositories/orgapachespark-1289
>> > >
>> > > The documentation corresponding to this release can be found at:
>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>> > >
>> > > The list of bug fixes 

Re: [VOTE] SPARK 2.4.0 (RC3)

[Dongjoon Hyun]

Yes. From my side, it's -1 for RC3.

Bests,
Dongjoon.

On Sat, Oct 13, 2018 at 1:24 PM Holden Karau  wrote:

> So if it's a blocker would you think this should be a -1?
>
> On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun 
> wrote:
>
>> Hi, Holden.
>>
>> Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Fri, Oct 12, 2018 at 11:45 AM Holden Karau 
>> wrote:
>>
>>> Following up I just wanted to make sure this new blocker that Dongjoon
>>> designated is surfaced -
>>> https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker
>>>
>>> On Thu, Oct 11, 2018 at 2:05 PM Xiao Li  wrote:
>>>
 -1. We have two correctness bugs:
 https://issues.apache.org/jira/browse/SPARK-25714 and
 https://issues.apache.org/jira/browse/SPARK-25708.

 Let us fix all the three issues in ScalaUDF, as mentioned by Sean.

 Xiao


 Sean Owen  于2018年10月11日周四 上午9:04写道：

> This is a legitimate question about the behavior of ScalaUDF after the
> change to support 2.12:
> https://github.com/apache/spark/pull/22259#discussion_r224295469
> Not quite a blocker I think, but a potential gotcha we definitely need
> to highlight in release notes. There may be an argument for changing
> ScalaUDF again before the release. Have a look, anyone familiar with
> catalyst.
> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen  wrote:
> >
> > +1. I tested the source build against Scala 2.12 and common build
> > profiles. License and sigs look OK.
> >
> > No blockers; one critical:
> >
> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
> >
> > I think this one is "won't fix" though? not trying to restore the
> behavior?
> >
> > Other items open for 2.4.0:
> >
> > SPARK-25347 Document image data source in doc site
> > SPARK-25584 Document libsvm data source in doc site
> > SPARK-25179 Document the features that require Pyarrow 0.10
> > SPARK-25507 Update documents for the new features in 2.4 release
> > SPARK-25346 Document Spark builtin data sources
> > SPARK-24464 Unit tests for MLlib's Instrumentation
> > SPARK-23197 Flaky test:
> spark.streaming.ReceiverSuite."receiver_life_cycle"
> > SPARK-22809 pyspark is sensitive to imports with dots
> > SPARK-21030 extend hint syntax to support any expression for Python
> and R
> >
> > Anyone know enough to close or retarget them? they don't look
> critical
> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document
> the
> > features that require Pyarrow 0.10" however sounds like it could have
> > been important for 2.4? if not a blocker.
> >
> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> > there is some ongoing discussion there.
> >
> > I am evaluating
> > https://github.com/apache/spark/pull/22259#discussion_r224252642
> right
> > now.
> >
> >
> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan 
> wrote:
> > >
> > > Please vote on releasing the following candidate as Apache Spark
> version 2.4.0.
> > >
> > > The vote is open until October 1 PST and passes if a majority +1
> PMC votes are cast, with
> > > a minimum of 3 +1 votes.
> > >
> > > [ ] +1 Release this package as Apache Spark 2.4.0
> > > [ ] -1 Do not release this package because ...
> > >
> > > To learn more about Apache Spark, please see
> http://spark.apache.org/
> > >
> > > The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > > https://github.com/apache/spark/tree/v2.4.0-rc3
> > >
> > > The release files, including signatures, digests, etc. can be
> found at:
> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> > >
> > > Signatures used for Spark RCs can be found in this file:
> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
> > >
> > > The staging repository for this release can be found at:
> > >
> https://repository.apache.org/content/repositories/orgapachespark-1289
> > >
> > > The documentation corresponding to this release can be found at:
> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> > >
> > > The list of bug fixes going into 2.4.0 can be found at the
> following URL:
> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> > >
> > > FAQ
> > >
> > > =
> > > How can I help test this release?
> > > =
> > >
> > > If you are a Spark user, you can help us test this release by

Re: [VOTE] SPARK 2.4.0 (RC3)

[Holden Karau]

So if it's a blocker would you think this should be a -1?

On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun 
wrote:

> Hi, Holden.
>
> Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.
>
> Bests,
> Dongjoon.
>
>
> On Fri, Oct 12, 2018 at 11:45 AM Holden Karau 
> wrote:
>
>> Following up I just wanted to make sure this new blocker that Dongjoon
>> designated is surfaced -
>> https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker
>>
>> On Thu, Oct 11, 2018 at 2:05 PM Xiao Li  wrote:
>>
>>> -1. We have two correctness bugs:
>>> https://issues.apache.org/jira/browse/SPARK-25714 and
>>> https://issues.apache.org/jira/browse/SPARK-25708.
>>>
>>> Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
>>>
>>> Xiao
>>>
>>>
>>> Sean Owen  于2018年10月11日周四 上午9:04写道：
>>>
 This is a legitimate question about the behavior of ScalaUDF after the
 change to support 2.12:
 https://github.com/apache/spark/pull/22259#discussion_r224295469
 Not quite a blocker I think, but a potential gotcha we definitely need
 to highlight in release notes. There may be an argument for changing
 ScalaUDF again before the release. Have a look, anyone familiar with
 catalyst.
 On Wed, Oct 10, 2018 at 3:00 PM Sean Owen  wrote:
 >
 > +1. I tested the source build against Scala 2.12 and common build
 > profiles. License and sigs look OK.
 >
 > No blockers; one critical:
 >
 > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
 >
 > I think this one is "won't fix" though? not trying to restore the
 behavior?
 >
 > Other items open for 2.4.0:
 >
 > SPARK-25347 Document image data source in doc site
 > SPARK-25584 Document libsvm data source in doc site
 > SPARK-25179 Document the features that require Pyarrow 0.10
 > SPARK-25507 Update documents for the new features in 2.4 release
 > SPARK-25346 Document Spark builtin data sources
 > SPARK-24464 Unit tests for MLlib's Instrumentation
 > SPARK-23197 Flaky test:
 spark.streaming.ReceiverSuite."receiver_life_cycle"
 > SPARK-22809 pyspark is sensitive to imports with dots
 > SPARK-21030 extend hint syntax to support any expression for Python
 and R
 >
 > Anyone know enough to close or retarget them? they don't look critical
 > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
 > features that require Pyarrow 0.10" however sounds like it could have
 > been important for 2.4? if not a blocker.
 >
 > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
 > there is some ongoing discussion there.
 >
 > I am evaluating
 > https://github.com/apache/spark/pull/22259#discussion_r224252642
 right
 > now.
 >
 >
 > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan 
 wrote:
 > >
 > > Please vote on releasing the following candidate as Apache Spark
 version 2.4.0.
 > >
 > > The vote is open until October 1 PST and passes if a majority +1
 PMC votes are cast, with
 > > a minimum of 3 +1 votes.
 > >
 > > [ ] +1 Release this package as Apache Spark 2.4.0
 > > [ ] -1 Do not release this package because ...
 > >
 > > To learn more about Apache Spark, please see
 http://spark.apache.org/
 > >
 > > The tag to be voted on is v2.4.0-rc3 (commit
 8e4a99bd201b9204fec52580f19ae70a229ed94e):
 > > https://github.com/apache/spark/tree/v2.4.0-rc3
 > >
 > > The release files, including signatures, digests, etc. can be found
 at:
 > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
 > >
 > > Signatures used for Spark RCs can be found in this file:
 > > https://dist.apache.org/repos/dist/dev/spark/KEYS
 > >
 > > The staging repository for this release can be found at:
 > >
 https://repository.apache.org/content/repositories/orgapachespark-1289
 > >
 > > The documentation corresponding to this release can be found at:
 > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
 > >
 > > The list of bug fixes going into 2.4.0 can be found at the
 following URL:
 > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
 > >
 > > FAQ
 > >
 > > =
 > > How can I help test this release?
 > > =
 > >
 > > If you are a Spark user, you can help us test this release by taking
 > > an existing Spark workload and running on this release candidate,
 then
 > > reporting any regressions.
 > >
 > > If you're working in PySpark you can set up a virtual env and
 install
 > > the current RC and see 

Re: [VOTE] SPARK 2.4.0 (RC3)

[Dongjoon Hyun]

Hi, Holden.

Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.

Bests,
Dongjoon.


On Fri, Oct 12, 2018 at 11:45 AM Holden Karau  wrote:

> Following up I just wanted to make sure this new blocker that Dongjoon
> designated is surfaced -
> https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker
>
> On Thu, Oct 11, 2018 at 2:05 PM Xiao Li  wrote:
>
>> -1. We have two correctness bugs:
>> https://issues.apache.org/jira/browse/SPARK-25714 and
>> https://issues.apache.org/jira/browse/SPARK-25708.
>>
>> Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
>>
>> Xiao
>>
>>
>> Sean Owen  于2018年10月11日周四 上午9:04写道：
>>
>>> This is a legitimate question about the behavior of ScalaUDF after the
>>> change to support 2.12:
>>> https://github.com/apache/spark/pull/22259#discussion_r224295469
>>> Not quite a blocker I think, but a potential gotcha we definitely need
>>> to highlight in release notes. There may be an argument for changing
>>> ScalaUDF again before the release. Have a look, anyone familiar with
>>> catalyst.
>>> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen  wrote:
>>> >
>>> > +1. I tested the source build against Scala 2.12 and common build
>>> > profiles. License and sigs look OK.
>>> >
>>> > No blockers; one critical:
>>> >
>>> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>>> >
>>> > I think this one is "won't fix" though? not trying to restore the
>>> behavior?
>>> >
>>> > Other items open for 2.4.0:
>>> >
>>> > SPARK-25347 Document image data source in doc site
>>> > SPARK-25584 Document libsvm data source in doc site
>>> > SPARK-25179 Document the features that require Pyarrow 0.10
>>> > SPARK-25507 Update documents for the new features in 2.4 release
>>> > SPARK-25346 Document Spark builtin data sources
>>> > SPARK-24464 Unit tests for MLlib's Instrumentation
>>> > SPARK-23197 Flaky test:
>>> spark.streaming.ReceiverSuite."receiver_life_cycle"
>>> > SPARK-22809 pyspark is sensitive to imports with dots
>>> > SPARK-21030 extend hint syntax to support any expression for Python
>>> and R
>>> >
>>> > Anyone know enough to close or retarget them? they don't look critical
>>> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
>>> > features that require Pyarrow 0.10" however sounds like it could have
>>> > been important for 2.4? if not a blocker.
>>> >
>>> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
>>> > there is some ongoing discussion there.
>>> >
>>> > I am evaluating
>>> > https://github.com/apache/spark/pull/22259#discussion_r224252642 right
>>> > now.
>>> >
>>> >
>>> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan 
>>> wrote:
>>> > >
>>> > > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.0.
>>> > >
>>> > > The vote is open until October 1 PST and passes if a majority +1 PMC
>>> votes are cast, with
>>> > > a minimum of 3 +1 votes.
>>> > >
>>> > > [ ] +1 Release this package as Apache Spark 2.4.0
>>> > > [ ] -1 Do not release this package because ...
>>> > >
>>> > > To learn more about Apache Spark, please see
>>> http://spark.apache.org/
>>> > >
>>> > > The tag to be voted on is v2.4.0-rc3 (commit
>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>> > > https://github.com/apache/spark/tree/v2.4.0-rc3
>>> > >
>>> > > The release files, including signatures, digests, etc. can be found
>>> at:
>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>> > >
>>> > > Signatures used for Spark RCs can be found in this file:
>>> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> > >
>>> > > The staging repository for this release can be found at:
>>> > >
>>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>> > >
>>> > > The documentation corresponding to this release can be found at:
>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>> > >
>>> > > The list of bug fixes going into 2.4.0 can be found at the following
>>> URL:
>>> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>> > >
>>> > > FAQ
>>> > >
>>> > > =
>>> > > How can I help test this release?
>>> > > =
>>> > >
>>> > > If you are a Spark user, you can help us test this release by taking
>>> > > an existing Spark workload and running on this release candidate,
>>> then
>>> > > reporting any regressions.
>>> > >
>>> > > If you're working in PySpark you can set up a virtual env and install
>>> > > the current RC and see if anything important breaks, in the
>>> Java/Scala
>>> > > you can add the staging repository to your projects resolvers and
>>> test
>>> > > with the RC (make sure to clean up the artifact cache before/after so
>>> > > you don't end up building with a 

Re: [VOTE] SPARK 2.4.0 (RC3)

[Xiao Li]

-1. We have two correctness bugs:
https://issues.apache.org/jira/browse/SPARK-25714 and
https://issues.apache.org/jira/browse/SPARK-25708.

Let us fix all the three issues in ScalaUDF, as mentioned by Sean.

Xiao


Sean Owen  于2018年10月11日周四 上午9:04写道：

> This is a legitimate question about the behavior of ScalaUDF after the
> change to support 2.12:
> https://github.com/apache/spark/pull/22259#discussion_r224295469
> Not quite a blocker I think, but a potential gotcha we definitely need
> to highlight in release notes. There may be an argument for changing
> ScalaUDF again before the release. Have a look, anyone familiar with
> catalyst.
> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen  wrote:
> >
> > +1. I tested the source build against Scala 2.12 and common build
> > profiles. License and sigs look OK.
> >
> > No blockers; one critical:
> >
> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
> >
> > I think this one is "won't fix" though? not trying to restore the
> behavior?
> >
> > Other items open for 2.4.0:
> >
> > SPARK-25347 Document image data source in doc site
> > SPARK-25584 Document libsvm data source in doc site
> > SPARK-25179 Document the features that require Pyarrow 0.10
> > SPARK-25507 Update documents for the new features in 2.4 release
> > SPARK-25346 Document Spark builtin data sources
> > SPARK-24464 Unit tests for MLlib's Instrumentation
> > SPARK-23197 Flaky test:
> spark.streaming.ReceiverSuite."receiver_life_cycle"
> > SPARK-22809 pyspark is sensitive to imports with dots
> > SPARK-21030 extend hint syntax to support any expression for Python and R
> >
> > Anyone know enough to close or retarget them? they don't look critical
> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> > features that require Pyarrow 0.10" however sounds like it could have
> > been important for 2.4? if not a blocker.
> >
> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> > there is some ongoing discussion there.
> >
> > I am evaluating
> > https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> > now.
> >
> >
> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan  wrote:
> > >
> > > Please vote on releasing the following candidate as Apache Spark
> version 2.4.0.
> > >
> > > The vote is open until October 1 PST and passes if a majority +1 PMC
> votes are cast, with
> > > a minimum of 3 +1 votes.
> > >
> > > [ ] +1 Release this package as Apache Spark 2.4.0
> > > [ ] -1 Do not release this package because ...
> > >
> > > To learn more about Apache Spark, please see http://spark.apache.org/
> > >
> > > The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > > https://github.com/apache/spark/tree/v2.4.0-rc3
> > >
> > > The release files, including signatures, digests, etc. can be found at:
> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> > >
> > > Signatures used for Spark RCs can be found in this file:
> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
> > >
> > > The staging repository for this release can be found at:
> > > https://repository.apache.org/content/repositories/orgapachespark-1289
> > >
> > > The documentation corresponding to this release can be found at:
> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> > >
> > > The list of bug fixes going into 2.4.0 can be found at the following
> URL:
> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> > >
> > > FAQ
> > >
> > > =
> > > How can I help test this release?
> > > =
> > >
> > > If you are a Spark user, you can help us test this release by taking
> > > an existing Spark workload and running on this release candidate, then
> > > reporting any regressions.
> > >
> > > If you're working in PySpark you can set up a virtual env and install
> > > the current RC and see if anything important breaks, in the Java/Scala
> > > you can add the staging repository to your projects resolvers and test
> > > with the RC (make sure to clean up the artifact cache before/after so
> > > you don't end up building with a out of date RC going forward).
> > >
> > > ===
> > > What should happen to JIRA tickets still targeting 2.4.0?
> > > ===
> > >
> > > The current list of open tickets targeted at 2.4.0 can be found at:
> > > https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
> > >
> > > Committers should look at those and triage. Extremely important bug
> > > fixes, documentation, and API tweaks that impact compatibility should
> > > be worked on immediately. Everything else please retarget to an
> > > appropriate release.
> > >
> > > ==
> > > But my bug isn't fixed?
> > > ==
> > >
> > > In order to make timely releases, we will typically not hold the
> > > release unless the bug in question is a 

Re: [VOTE] SPARK 2.4.0 (RC3)

[Sean Owen]

This is a legitimate question about the behavior of ScalaUDF after the
change to support 2.12:
https://github.com/apache/spark/pull/22259#discussion_r224295469
Not quite a blocker I think, but a potential gotcha we definitely need
to highlight in release notes. There may be an argument for changing
ScalaUDF again before the release. Have a look, anyone familiar with
catalyst.
On Wed, Oct 10, 2018 at 3:00 PM Sean Owen  wrote:
>
> +1. I tested the source build against Scala 2.12 and common build
> profiles. License and sigs look OK.
>
> No blockers; one critical:
>
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>
> I think this one is "won't fix" though? not trying to restore the behavior?
>
> Other items open for 2.4.0:
>
> SPARK-25347 Document image data source in doc site
> SPARK-25584 Document libsvm data source in doc site
> SPARK-25179 Document the features that require Pyarrow 0.10
> SPARK-25507 Update documents for the new features in 2.4 release
> SPARK-25346 Document Spark builtin data sources
> SPARK-24464 Unit tests for MLlib's Instrumentation
> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
> SPARK-22809 pyspark is sensitive to imports with dots
> SPARK-21030 extend hint syntax to support any expression for Python and R
>
> Anyone know enough to close or retarget them? they don't look critical
> for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> features that require Pyarrow 0.10" however sounds like it could have
> been important for 2.4? if not a blocker.
>
> PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> there is some ongoing discussion there.
>
> I am evaluating
> https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> now.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan  wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version 
> > 2.4.0.
> >
> > The vote is open until October 1 PST and passes if a majority +1 PMC votes 
> > are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.0-rc3 (commit 
> > 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > https://github.com/apache/spark/tree/v2.4.0-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1289
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> >
> > The list of bug fixes going into 2.4.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> >
> > FAQ
> >
> > =
> > How can I help test this release?
> > =
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===
> > What should happen to JIRA tickets still targeting 2.4.0?
> > ===
> >
> > The current list of open tickets targeted at 2.4.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target 
> > Version/s" = 2.4.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==
> > But my bug isn't fixed?
> > ==
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] SPARK 2.4.0 (RC3)

[Hyukjin Kwon]

So, which date is it?

2018년 10월 11일 (목) 오전 1:48, Garlapati, Suryanarayana (Nokia - IN/Bangalore) <
suryanarayana.garlap...@nokia.com>님이 작성:

> Might be you need to change the date(Oct 1 has already passed).
>
>
>
> >> The vote is open until October 1 PST and passes if a majority +1 PMC
> votes are cast, with
>
> >> a minimum of 3 +1 votes.
>
>
>
> Regards
>
> Surya
>
>
>
> *From:* Wenchen Fan 
> *Sent:* Wednesday, October 10, 2018 10:20 PM
> *To:* Spark dev list 
> *Subject:* Re: [VOTE] SPARK 2.4.0 (RC3)
>
>
>
> I'm adding my own +1, since there are no known blocker issues. The
> correctness issue has been fixed, the streaming Java API problem has been
> resolved, and we have upgraded to Scala 2.12.7.
>
>
>
> On Thu, Oct 11, 2018 at 12:46 AM Wenchen Fan  wrote:
>
> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
>
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
>
> a minimum of 3 +1 votes.
>
>
>
> [ ] +1 Release this package as Apache Spark 2.4.0
>
> [ ] -1 Do not release this package because ...
>
>
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
>
>
> The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
>
>
> The release files, including signatures, digests, etc. can be found at:
>
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
>
>
> Signatures used for Spark RCs can be found in this file:
>
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
>
>
> The staging repository for this release can be found at:
>
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
>
>
> The documentation corresponding to this release can be found at:
>
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
>
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
>
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
>
>
> FAQ
>
>
>
> =
>
> How can I help test this release?
>
> =
>
>
>
> If you are a Spark user, you can help us test this release by taking
>
> an existing Spark workload and running on this release candidate, then
>
> reporting any regressions.
>
>
>
> If you're working in PySpark you can set up a virtual env and install
>
> the current RC and see if anything important breaks, in the Java/Scala
>
> you can add the staging repository to your projects resolvers and test
>
> with the RC (make sure to clean up the artifact cache before/after so
>
> you don't end up building with a out of date RC going forward).
>
>
>
> ===
>
> What should happen to JIRA tickets still targeting 2.4.0?
>
> ===
>
>
>
> The current list of open tickets targeted at 2.4.0 can be found at:
>
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
>
>
> Committers should look at those and triage. Extremely important bug
>
> fixes, documentation, and API tweaks that impact compatibility should
>
> be worked on immediately. Everything else please retarget to an
>
> appropriate release.
>
>
>
> ==
>
> But my bug isn't fixed?
>
> ==
>
>
>
> In order to make timely releases, we will typically not hold the
>
> release unless the bug in question is a regression from the previous
>
> release. That being said, if there is something which is a regression
>
> that has not been correctly targeted please ping me or a committer to
>
> help target the issue.
>
>

Re: [VOTE] SPARK 2.4.0 (RC3)

[Sean Owen]

+1. I tested the source build against Scala 2.12 and common build
profiles. License and sigs look OK.

No blockers; one critical:

SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4

I think this one is "won't fix" though? not trying to restore the behavior?

Other items open for 2.4.0:

SPARK-25347 Document image data source in doc site
SPARK-25584 Document libsvm data source in doc site
SPARK-25179 Document the features that require Pyarrow 0.10
SPARK-25507 Update documents for the new features in 2.4 release
SPARK-25346 Document Spark builtin data sources
SPARK-24464 Unit tests for MLlib's Instrumentation
SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
SPARK-22809 pyspark is sensitive to imports with dots
SPARK-21030 extend hint syntax to support any expression for Python and R

Anyone know enough to close or retarget them? they don't look critical
for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
features that require Pyarrow 0.10" however sounds like it could have
been important for 2.4? if not a blocker.

PS I don't think that SPARK-25150 is an issue; see JIRA. At least
there is some ongoing discussion there.

I am evaluating
https://github.com/apache/spark/pull/22259#discussion_r224252642 right
now.


On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan  wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes 
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc3 (commit 
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.0?
> ===
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target 
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] SPARK 2.4.0 (RC3)

[Nicholas Chammas]

FYI I believe we have an open correctness issue here:

https://issues.apache.org/jira/browse/SPARK-25150

However, it needs review by another person to confirm whether it is indeed
a correctness issue (and whether it still impacts this latest RC).

Nick

2018년 10월 10일 (수) 오후 3:14, Jean Georges Perrin 님이 작성:

> Awesome - thanks Dongjoon!
>
>
> On Oct 10, 2018, at 2:36 PM, Dongjoon Hyun 
> wrote:
>
> For now, you can see generated release notes. Official one will be posted
> on the website when the official 2.4.0 is out.
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420=12342385
>
> Bests,
> Dongjoon.
>
>
> On Wed, Oct 10, 2018 at 11:29 AM Jean Georges Perrin  wrote:
>
>> Hi,
>>
>> Sorry if it's stupid question, but where can I find the release notes of
>> 2.4.0?
>>
>> jg
>>
>> On Oct 10, 2018, at 2:00 PM, Imran Rashid 
>> wrote:
>>
>> Sorry I had messed up my testing earlier, so I only just discovered
>> https://issues.apache.org/jira/browse/SPARK-25704
>>
>> I dont' think this is a release blocker, because its not a regression and
>> there is a workaround, just fyi.
>>
>> On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan  wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 2.4.0.
>>>
>>> The vote is open until October 1 PST and passes if a majority +1 PMC
>>> votes are cast, with
>>> a minimum of 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.4.0
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v2.4.0-rc3 (commit
>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>> https://github.com/apache/spark/tree/v2.4.0-rc3
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>>
>>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>>> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>>
>>> FAQ
>>>
>>> =
>>> How can I help test this release?
>>> =
>>>
>>> If you are a Spark user, you can help us test this release by taking
>>> an existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>> the current RC and see if anything important breaks, in the Java/Scala
>>> you can add the staging repository to your projects resolvers and test
>>> with the RC (make sure to clean up the artifact cache before/after so
>>> you don't end up building with a out of date RC going forward).
>>>
>>> ===
>>> What should happen to JIRA tickets still targeting 2.4.0?
>>> ===
>>>
>>> The current list of open tickets targeted at 2.4.0 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.0
>>>
>>> Committers should look at those and triage. Extremely important bug
>>> fixes, documentation, and API tweaks that impact compatibility should
>>> be worked on immediately. Everything else please retarget to an
>>> appropriate release.
>>>
>>> ==
>>> But my bug isn't fixed?
>>> ==
>>>
>>> In order to make timely releases, we will typically not hold the
>>> release unless the bug in question is a regression from the previous
>>> release. That being said, if there is something which is a regression
>>> that has not been correctly targeted please ping me or a committer to
>>> help target the issue.
>>>
>>
>>
>

Re: [VOTE] SPARK 2.4.0 (RC3)

[Jean Georges Perrin]

Awesome - thanks Dongjoon!

> On Oct 10, 2018, at 2:36 PM, Dongjoon Hyun  wrote:
> 
> For now, you can see generated release notes. Official one will be posted on 
> the website when the official 2.4.0 is out.
> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420=12342385
>  
> 
> 
> Bests,
> Dongjoon.
> 
> 
> On Wed, Oct 10, 2018 at 11:29 AM Jean Georges Perrin  > wrote:
> Hi,
> 
> Sorry if it's stupid question, but where can I find the release notes of 
> 2.4.0?
> 
> jg
> 
>> On Oct 10, 2018, at 2:00 PM, Imran Rashid > > wrote:
>> 
>> Sorry I had messed up my testing earlier, so I only just discovered 
>> https://issues.apache.org/jira/browse/SPARK-25704 
>> 
>> 
>> I dont' think this is a release blocker, because its not a regression and 
>> there is a workaround, just fyi.
>> 
>> On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan > > wrote:
>> Please vote on releasing the following candidate as Apache Spark version 
>> 2.4.0.
>> 
>> The vote is open until October 1 PST and passes if a majority +1 PMC votes 
>> are cast, with
>> a minimum of 3 +1 votes.
>> 
>> [ ] +1 Release this package as Apache Spark 2.4.0
>> [ ] -1 Do not release this package because ...
>> 
>> To learn more about Apache Spark, please see http://spark.apache.org/ 
>> 
>> 
>> The tag to be voted on is v2.4.0-rc3 (commit 
>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>> https://github.com/apache/spark/tree/v2.4.0-rc3 
>> 
>> 
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/ 
>> 
>> 
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS 
>> 
>> 
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1289 
>> 
>> 
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/ 
>> 
>> 
>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12342385 
>> 
>> 
>> FAQ
>> 
>> =
>> How can I help test this release?
>> =
>> 
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>> 
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>> 
>> ===
>> What should happen to JIRA tickets still targeting 2.4.0?
>> ===
>> 
>> The current list of open tickets targeted at 2.4.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK 
>>  and search for "Target 
>> Version/s" = 2.4.0
>> 
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>> 
>> ==
>> But my bug isn't fixed?
>> ==
>> 
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
> 

Re: [VOTE] SPARK 2.4.0 (RC3)

[Dongjoon Hyun]

For now, you can see generated release notes. Official one will be posted
on the website when the official 2.4.0 is out.

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420=12342385

Bests,
Dongjoon.


On Wed, Oct 10, 2018 at 11:29 AM Jean Georges Perrin  wrote:

> Hi,
>
> Sorry if it's stupid question, but where can I find the release notes of
> 2.4.0?
>
> jg
>
> On Oct 10, 2018, at 2:00 PM, Imran Rashid 
> wrote:
>
> Sorry I had messed up my testing earlier, so I only just discovered
> https://issues.apache.org/jira/browse/SPARK-25704
>
> I dont' think this is a release blocker, because its not a regression and
> there is a workaround, just fyi.
>
> On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan  wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 2.4.0.
>>
>> The vote is open until October 1 PST and passes if a majority +1 PMC
>> votes are cast, with
>> a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 2.4.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v2.4.0-rc3 (commit
>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>> https://github.com/apache/spark/tree/v2.4.0-rc3
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>
>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>
>> FAQ
>>
>> =
>> How can I help test this release?
>> =
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===
>> What should happen to JIRA tickets still targeting 2.4.0?
>> ===
>>
>> The current list of open tickets targeted at 2.4.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==
>> But my bug isn't fixed?
>> ==
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>
>

Re: [VOTE] SPARK 2.4.0 (RC3)

[Jean Georges Perrin]

Hi,

Sorry if it's stupid question, but where can I find the release notes of 2.4.0?

jg

> On Oct 10, 2018, at 2:00 PM, Imran Rashid  > wrote:
> 
> Sorry I had messed up my testing earlier, so I only just discovered 
> https://issues.apache.org/jira/browse/SPARK-25704 
> 
> 
> I dont' think this is a release blocker, because its not a regression and 
> there is a workaround, just fyi.
> 
> On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan  > wrote:
> Please vote on releasing the following candidate as Apache Spark version 
> 2.4.0.
> 
> The vote is open until October 1 PST and passes if a majority +1 PMC votes 
> are cast, with
> a minimum of 3 +1 votes.
> 
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
> 
> To learn more about Apache Spark, please see http://spark.apache.org/ 
> 
> 
> The tag to be voted on is v2.4.0-rc3 (commit 
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3 
> 
> 
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/ 
> 
> 
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS 
> 
> 
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289 
> 
> 
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/ 
> 
> 
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385 
> 
> 
> FAQ
> 
> =
> How can I help test this release?
> =
> 
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
> 
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
> 
> ===
> What should happen to JIRA tickets still targeting 2.4.0?
> ===
> 
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK 
>  and search for "Target 
> Version/s" = 2.4.0
> 
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
> 
> ==
> But my bug isn't fixed?
> ==
> 
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

Re: [VOTE] SPARK 2.4.0 (RC3)

[Imran Rashid]

Sorry I had messed up my testing earlier, so I only just discovered
https://issues.apache.org/jira/browse/SPARK-25704

I dont' think this is a release blocker, because its not a regression and
there is a workaround, just fyi.

On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.0?
> ===
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>

RE: [VOTE] SPARK 2.4.0 (RC3)

[Garlapati, Suryanarayana (Nokia - IN/Bangalore)]

Might be you need to change the date(Oct 1 has already passed).

>> The vote is open until October 1 PST and passes if a majority +1 PMC votes 
>> are cast, with
>> a minimum of 3 +1 votes.

Regards
Surya

From: Wenchen Fan 
Sent: Wednesday, October 10, 2018 10:20 PM
To: Spark dev list 
Subject: Re: [VOTE] SPARK 2.4.0 (RC3)

I'm adding my own +1, since there are no known blocker issues. The correctness 
issue has been fixed, the streaming Java API problem has been resolved, and we 
have upgraded to Scala 2.12.7.

On Thu, Oct 11, 2018 at 12:46 AM Wenchen Fan 
mailto:cloud0...@gmail.com>> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are 
cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc3 (commit 
8e4a99bd201b9204fec52580f19ae70a229ed94e):
https://github.com/apache/spark/tree/v2.4.0-rc3

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1289

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/

The list of bug fixes going into 2.4.0 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12342385

FAQ

=
How can I help test this release?
=

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===
What should happen to JIRA tickets still targeting 2.4.0?
===

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" 
= 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==
But my bug isn't fixed?
==

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Re: [VOTE] SPARK 2.4.0 (RC3)

[Wenchen Fan]

I'm adding my own +1, since there are no known blocker issues. The
correctness issue has been fixed, the streaming Java API problem has been
resolved, and we have upgraded to Scala 2.12.7.

On Thu, Oct 11, 2018 at 12:46 AM Wenchen Fan  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.0?
> ===
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>