Re: Make Scala 2.12 as default Scala version in Spark 3.0

[Sean Owen]

That's possible here, sure. The issue is: would you exclude Scala 2.13
support in 3.0 for this, if it were otherwise ready to go?
I think it's not a hard rule that something has to be deprecated
previously to be removed in a major release. The notice is helpful,
sure, but there are lots of ways to provide that notice to end users.
Lots of things are breaking changes in a major release. Or: deprecate
in Spark 2.4.1, if desired?

On Tue, Nov 6, 2018 at 7:36 PM Wenchen Fan  wrote:
>
> We make Scala 2.11 the default one in Spark 2.0, then drop Scala 2.10 in 
> Spark 2.3. Shall we follow it and drop Scala 2.11 at some point of Spark 3.x?
>
> On Wed, Nov 7, 2018 at 8:55 AM Reynold Xin  wrote:
>>
>> Have we deprecated Scala 2.11 already in an existing release?

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Make Scala 2.12 as default Scala version in Spark 3.0

[Wenchen Fan]

We make Scala 2.11 the default one in Spark 2.0, then drop Scala 2.10 in
Spark 2.3. Shall we follow it and drop Scala 2.11 at some point of Spark
3.x?

On Wed, Nov 7, 2018 at 8:55 AM Reynold Xin  wrote:

> Have we deprecated Scala 2.11 already in an existing release?
>
> On Tue, Nov 6, 2018 at 4:43 PM DB Tsai  wrote:
>
>> Ideally, supporting only Scala 2.12 in Spark 3 will be ideal.
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
>> Apple, Inc
>>
>> > On Nov 6, 2018, at 2:55 PM, Felix Cheung 
>> wrote:
>> >
>> > So to clarify, only scala 2.12 is supported in Spark 3?
>> >
>> >
>> > From: Ryan Blue 
>> > Sent: Tuesday, November 6, 2018 1:24 PM
>> > To: d_t...@apple.com
>> > Cc: Sean Owen; Spark Dev List; cdelg...@apple.com
>> > Subject: Re: Make Scala 2.12 as default Scala version in Spark 3.0
>> >
>> > +1 to Scala 2.12 as the default in Spark 3.0.
>> >
>> > On Tue, Nov 6, 2018 at 11:50 AM DB Tsai  wrote:
>> > +1 on dropping Scala 2.11 in Spark 3.0 to simplify the build.
>> >
>> > As Scala 2.11 will not support Java 11 unless we make a significant
>> investment, if we decide not to drop Scala 2.11 in Spark 3.0, what we can
>> do is have only Scala 2.12 build support Java 11 while Scala 2.11 support
>> Java 8. But I agree with Sean that this can make the decencies really
>> complicated; hence I support to drop Scala 2.11 in Spark 3.0 directly.
>> >
>> > DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
>> Apple, Inc
>> >
>> >> On Nov 6, 2018, at 11:38 AM, Sean Owen  wrote:
>> >>
>> >> I think we should make Scala 2.12 the default in Spark 3.0. I would
>> >> also prefer to drop Scala 2.11 support in 3.0. In theory, not dropping
>> >> 2.11 support it means we'd support Scala 2.11 for years, the lifetime
>> >> of Spark 3.x. In practice, we could drop 2.11 support in a 3.1.0 or
>> >> 3.2.0 release, kind of like what happened with 2.10 in 2.x.
>> >>
>> >> Java (9-)11 support also complicates this. I think getting it to work
>> >> will need some significant dependency updates, and I worry not all
>> >> will be available for 2.11 or will present some knotty problems. We'll
>> >> find out soon if that forces the issue.
>> >>
>> >> Also note that Scala 2.13 is pretty close to release, and we'll want
>> >> to support it soon after release, perhaps sooner than the long delay
>> >> before 2.12 was supported (because it was hard!). It will probably be
>> >> out well before Spark 3.0. Cross-compiling for 3 Scala versions sounds
>> >> like too much. 3.0 could support 2.11 and 2.12, and 3.1 support 2.12
>> >> and 2.13, or something. But if 2.13 support is otherwise attainable at
>> >> the release of Spark 3.0, I wonder if that too argues for dropping
>> >> 2.11 support.
>> >>
>> >> Finally I'll say that Spark itself isn't dropping 2.11 support for a
>> >> while, no matter what; it still exists in the 2.4.x branch of course.
>> >> People who can't update off Scala 2.11 can stay on Spark 2.x, note.
>> >>
>> >> Sean
>> >>
>> >>
>> >> On Tue, Nov 6, 2018 at 1:13 PM DB Tsai  wrote:
>> >>>
>> >>> We made Scala 2.11 as default Scala version in Spark 2.0. Now, the
>> next Spark version will be 3.0, so it's a great time to discuss should we
>> make Scala 2.12 as default Scala version in Spark 3.0.
>> >>>
>> >>> Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's
>> unlikely to support JDK 11 in Scala 2.11 unless we're willing to sponsor
>> the needed work per discussion in Scala community,
>> https://github.com/scala/scala-dev/issues/559#issuecomment-436160166
>> >>>
>> >>> We have initial support of Scala 2.12 in Spark 2.4. If we decide to
>> make Scala 2.12 as default for Spark 3.0 now, we will have ample time to
>> work on bugs and issues that we may run into.
>> >>>
>> >>> What do you think?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
>> Apple, Inc
>> >>>
>> >>>
>> >>> -
>> >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >>>
>> >
>> >
>> >
>> > --
>> > Ryan Blue
>> > Software Engineer
>> > Netflix
>>
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: Make Scala 2.12 as default Scala version in Spark 3.0

[Reynold Xin]

Have we deprecated Scala 2.11 already in an existing release?

On Tue, Nov 6, 2018 at 4:43 PM DB Tsai  wrote:

> Ideally, supporting only Scala 2.12 in Spark 3 will be ideal.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
>
> > On Nov 6, 2018, at 2:55 PM, Felix Cheung 
> wrote:
> >
> > So to clarify, only scala 2.12 is supported in Spark 3?
> >
> >
> > From: Ryan Blue 
> > Sent: Tuesday, November 6, 2018 1:24 PM
> > To: d_t...@apple.com
> > Cc: Sean Owen; Spark Dev List; cdelg...@apple.com
> > Subject: Re: Make Scala 2.12 as default Scala version in Spark 3.0
> >
> > +1 to Scala 2.12 as the default in Spark 3.0.
> >
> > On Tue, Nov 6, 2018 at 11:50 AM DB Tsai  wrote:
> > +1 on dropping Scala 2.11 in Spark 3.0 to simplify the build.
> >
> > As Scala 2.11 will not support Java 11 unless we make a significant
> investment, if we decide not to drop Scala 2.11 in Spark 3.0, what we can
> do is have only Scala 2.12 build support Java 11 while Scala 2.11 support
> Java 8. But I agree with Sean that this can make the decencies really
> complicated; hence I support to drop Scala 2.11 in Spark 3.0 directly.
> >
> > DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
> >
> >> On Nov 6, 2018, at 11:38 AM, Sean Owen  wrote:
> >>
> >> I think we should make Scala 2.12 the default in Spark 3.0. I would
> >> also prefer to drop Scala 2.11 support in 3.0. In theory, not dropping
> >> 2.11 support it means we'd support Scala 2.11 for years, the lifetime
> >> of Spark 3.x. In practice, we could drop 2.11 support in a 3.1.0 or
> >> 3.2.0 release, kind of like what happened with 2.10 in 2.x.
> >>
> >> Java (9-)11 support also complicates this. I think getting it to work
> >> will need some significant dependency updates, and I worry not all
> >> will be available for 2.11 or will present some knotty problems. We'll
> >> find out soon if that forces the issue.
> >>
> >> Also note that Scala 2.13 is pretty close to release, and we'll want
> >> to support it soon after release, perhaps sooner than the long delay
> >> before 2.12 was supported (because it was hard!). It will probably be
> >> out well before Spark 3.0. Cross-compiling for 3 Scala versions sounds
> >> like too much. 3.0 could support 2.11 and 2.12, and 3.1 support 2.12
> >> and 2.13, or something. But if 2.13 support is otherwise attainable at
> >> the release of Spark 3.0, I wonder if that too argues for dropping
> >> 2.11 support.
> >>
> >> Finally I'll say that Spark itself isn't dropping 2.11 support for a
> >> while, no matter what; it still exists in the 2.4.x branch of course.
> >> People who can't update off Scala 2.11 can stay on Spark 2.x, note.
> >>
> >> Sean
> >>
> >>
> >> On Tue, Nov 6, 2018 at 1:13 PM DB Tsai  wrote:
> >>>
> >>> We made Scala 2.11 as default Scala version in Spark 2.0. Now, the
> next Spark version will be 3.0, so it's a great time to discuss should we
> make Scala 2.12 as default Scala version in Spark 3.0.
> >>>
> >>> Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely
> to support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed
> work per discussion in Scala community,
> https://github.com/scala/scala-dev/issues/559#issuecomment-436160166
> >>>
> >>> We have initial support of Scala 2.12 in Spark 2.4. If we decide to
> make Scala 2.12 as default for Spark 3.0 now, we will have ample time to
> work on bugs and issues that we may run into.
> >>>
> >>> What do you think?
> >>>
> >>> Thanks,
> >>>
> >>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
> >>>
> >>>
> >>> -
> >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >>>
> >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: Make Scala 2.12 as default Scala version in Spark 3.0

[DB Tsai]

Ideally, supporting only Scala 2.12 in Spark 3 will be ideal.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Nov 6, 2018, at 2:55 PM, Felix Cheung  wrote:
> 
> So to clarify, only scala 2.12 is supported in Spark 3?
> 
>  
> From: Ryan Blue 
> Sent: Tuesday, November 6, 2018 1:24 PM
> To: d_t...@apple.com
> Cc: Sean Owen; Spark Dev List; cdelg...@apple.com
> Subject: Re: Make Scala 2.12 as default Scala version in Spark 3.0
>  
> +1 to Scala 2.12 as the default in Spark 3.0.
> 
> On Tue, Nov 6, 2018 at 11:50 AM DB Tsai  wrote:
> +1 on dropping Scala 2.11 in Spark 3.0 to simplify the build. 
> 
> As Scala 2.11 will not support Java 11 unless we make a significant 
> investment, if we decide not to drop Scala 2.11 in Spark 3.0, what we can do 
> is have only Scala 2.12 build support Java 11 while Scala 2.11 support Java 
> 8. But I agree with Sean that this can make the decencies really complicated; 
> hence I support to drop Scala 2.11 in Spark 3.0 directly.
> 
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
> Inc
> 
>> On Nov 6, 2018, at 11:38 AM, Sean Owen  wrote:
>> 
>> I think we should make Scala 2.12 the default in Spark 3.0. I would
>> also prefer to drop Scala 2.11 support in 3.0. In theory, not dropping
>> 2.11 support it means we'd support Scala 2.11 for years, the lifetime
>> of Spark 3.x. In practice, we could drop 2.11 support in a 3.1.0 or
>> 3.2.0 release, kind of like what happened with 2.10 in 2.x.
>> 
>> Java (9-)11 support also complicates this. I think getting it to work
>> will need some significant dependency updates, and I worry not all
>> will be available for 2.11 or will present some knotty problems. We'll
>> find out soon if that forces the issue.
>> 
>> Also note that Scala 2.13 is pretty close to release, and we'll want
>> to support it soon after release, perhaps sooner than the long delay
>> before 2.12 was supported (because it was hard!). It will probably be
>> out well before Spark 3.0. Cross-compiling for 3 Scala versions sounds
>> like too much. 3.0 could support 2.11 and 2.12, and 3.1 support 2.12
>> and 2.13, or something. But if 2.13 support is otherwise attainable at
>> the release of Spark 3.0, I wonder if that too argues for dropping
>> 2.11 support.
>> 
>> Finally I'll say that Spark itself isn't dropping 2.11 support for a
>> while, no matter what; it still exists in the 2.4.x branch of course.
>> People who can't update off Scala 2.11 can stay on Spark 2.x, note.
>> 
>> Sean
>> 
>> 
>> On Tue, Nov 6, 2018 at 1:13 PM DB Tsai  wrote:
>>> 
>>> We made Scala 2.11 as default Scala version in Spark 2.0. Now, the next 
>>> Spark version will be 3.0, so it's a great time to discuss should we make 
>>> Scala 2.12 as default Scala version in Spark 3.0.
>>> 
>>> Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely to 
>>> support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed 
>>> work per discussion in Scala community, 
>>> https://github.com/scala/scala-dev/issues/559#issuecomment-436160166
>>> 
>>> We have initial support of Scala 2.12 in Spark 2.4. If we decide to make 
>>> Scala 2.12 as default for Spark 3.0 now, we will have ample time to work on 
>>> bugs and issues that we may run into.
>>> 
>>> What do you think?
>>> 
>>> Thanks,
>>> 
>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
>>> Inc
>>> 
>>> 
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>> 
> 
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Netflix


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Test and support only LTS JDK release?

[Marcelo Vanzin]

https://www.oracle.com/technetwork/java/javase/eol-135779.html
On Tue, Nov 6, 2018 at 2:56 PM Felix Cheung  wrote:
>
> Is there a list of LTS release that I can reference?
>
>
> 
> From: Ryan Blue 
> Sent: Tuesday, November 6, 2018 1:28 PM
> To: sn...@snazy.de
> Cc: Spark Dev List; cdelg...@apple.com
> Subject: Re: Test and support only LTS JDK release?
>
> +1 for supporting LTS releases.
>
> On Tue, Nov 6, 2018 at 11:48 AM Robert Stupp  wrote:
>>
>> +1 on supporting LTS releases.
>>
>> VM distributors (RedHat, Azul - to name two) want to provide patches to LTS 
>> versions (i.e. into http://hg.openjdk.java.net/jdk-updates/jdk11u/). How 
>> that will play out in reality ... I don't know. Whether Oracle will 
>> contribute to that repo for 8 after it's EOL and 11 after the 6 month cycle 
>> ... we will see. Most Linux distributions promised(?) long-term support for 
>> Java 11 in their LTS releases (e.g. Ubuntu 18.04). I am not sure what that 
>> exactly means ... whether they will actively provide patches to OpenJDK or 
>> whether they just build from source.
>>
>> But considering that, I think it's definitely worth to at least keep an eye 
>> on Java 12 and 13 - even if those are just EA. Java 12 for example does 
>> already forbid some "dirty tricks" that are still possible in Java 11.
>>
>>
>> On 11/6/18 8:32 PM, DB Tsai wrote:
>>
>> OpenJDK will follow Oracle's release cycle, 
>> https://openjdk.java.net/projects/jdk/, a strict six months model. I'm not 
>> familiar with other non-Oracle VMs and Redhat support.
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
>> Inc
>>
>> On Nov 6, 2018, at 11:26 AM, Reynold Xin  wrote:
>>
>> What does OpenJDK do and other non-Oracle VMs? I know there was a lot of 
>> discussions from Redhat etc to support.
>>
>>
>> On Tue, Nov 6, 2018 at 11:24 AM DB Tsai  wrote:
>>>
>>> Given Oracle's new 6-month release model, I feel the only realistic option 
>>> is to only test and support JDK such as JDK 11 LTS and future LTS release. 
>>> I would like to have a discussion on this in Spark community.
>>>
>>> Thanks,
>>>
>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
>>> Inc
>>>
>>
>> --
>> Robert Stupp
>> @snazy
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix



-- 
Marcelo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Test and support only LTS JDK release?

[Felix Cheung]

Is there a list of LTS release that I can reference?



From: Ryan Blue 
Sent: Tuesday, November 6, 2018 1:28 PM
To: sn...@snazy.de
Cc: Spark Dev List; cdelg...@apple.com
Subject: Re: Test and support only LTS JDK release?

+1 for supporting LTS releases.

On Tue, Nov 6, 2018 at 11:48 AM Robert Stupp 
mailto:sn...@snazy.de>> wrote:

+1 on supporting LTS releases.

VM distributors (RedHat, Azul - to name two) want to provide patches to LTS 
versions (i.e. into http://hg.openjdk.java.net/jdk-updates/jdk11u/). How that 
will play out in reality ... I don't know. Whether Oracle will contribute to 
that repo for 8 after it's EOL and 11 after the 6 month cycle ... we will see. 
Most Linux distributions promised(?) long-term support for Java 11 in their LTS 
releases (e.g. Ubuntu 18.04). I am not sure what that exactly means ... whether 
they will actively provide patches to OpenJDK or whether they just build from 
source.

But considering that, I think it's definitely worth to at least keep an eye on 
Java 12 and 13 - even if those are just EA. Java 12 for example does already 
forbid some "dirty tricks" that are still possible in Java 11.


On 11/6/18 8:32 PM, DB Tsai wrote:
OpenJDK will follow Oracle's release cycle, 
https://openjdk.java.net/projects/jdk/, a strict six months model. I'm not 
familiar with other non-Oracle VMs and Redhat support.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

On Nov 6, 2018, at 11:26 AM, Reynold Xin 
mailto:r...@databricks.com>> wrote:

What does OpenJDK do and other non-Oracle VMs? I know there was a lot of 
discussions from Redhat etc to support.


On Tue, Nov 6, 2018 at 11:24 AM DB Tsai 
mailto:d_t...@apple.com>> wrote:
Given Oracle's new 6-month release model, I feel the only realistic option is 
to only test and support JDK such as JDK 11 LTS and future LTS release. I would 
like to have a discussion on this in Spark community.

Thanks,

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc



--
Robert Stupp
@snazy


--
Ryan Blue
Software Engineer
Netflix

Re: Make Scala 2.12 as default Scala version in Spark 3.0

[Felix Cheung]

So to clarify, only scala 2.12 is supported in Spark 3?



From: Ryan Blue 
Sent: Tuesday, November 6, 2018 1:24 PM
To: d_t...@apple.com
Cc: Sean Owen; Spark Dev List; cdelg...@apple.com
Subject: Re: Make Scala 2.12 as default Scala version in Spark 3.0

+1 to Scala 2.12 as the default in Spark 3.0.

On Tue, Nov 6, 2018 at 11:50 AM DB Tsai 
mailto:d_t...@apple.com>> wrote:
+1 on dropping Scala 2.11 in Spark 3.0 to simplify the build.

As Scala 2.11 will not support Java 11 unless we make a significant investment, 
if we decide not to drop Scala 2.11 in Spark 3.0, what we can do is have only 
Scala 2.12 build support Java 11 while Scala 2.11 support Java 8. But I agree 
with Sean that this can make the decencies really complicated; hence I support 
to drop Scala 2.11 in Spark 3.0 directly.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

On Nov 6, 2018, at 11:38 AM, Sean Owen 
mailto:sro...@gmail.com>> wrote:

I think we should make Scala 2.12 the default in Spark 3.0. I would
also prefer to drop Scala 2.11 support in 3.0. In theory, not dropping
2.11 support it means we'd support Scala 2.11 for years, the lifetime
of Spark 3.x. In practice, we could drop 2.11 support in a 3.1.0 or
3.2.0 release, kind of like what happened with 2.10 in 2.x.

Java (9-)11 support also complicates this. I think getting it to work
will need some significant dependency updates, and I worry not all
will be available for 2.11 or will present some knotty problems. We'll
find out soon if that forces the issue.

Also note that Scala 2.13 is pretty close to release, and we'll want
to support it soon after release, perhaps sooner than the long delay
before 2.12 was supported (because it was hard!). It will probably be
out well before Spark 3.0. Cross-compiling for 3 Scala versions sounds
like too much. 3.0 could support 2.11 and 2.12, and 3.1 support 2.12
and 2.13, or something. But if 2.13 support is otherwise attainable at
the release of Spark 3.0, I wonder if that too argues for dropping
2.11 support.

Finally I'll say that Spark itself isn't dropping 2.11 support for a
while, no matter what; it still exists in the 2.4.x branch of course.
People who can't update off Scala 2.11 can stay on Spark 2.x, note.

Sean


On Tue, Nov 6, 2018 at 1:13 PM DB Tsai 
mailto:d_t...@apple.com>> wrote:

We made Scala 2.11 as default Scala version in Spark 2.0. Now, the next Spark 
version will be 3.0, so it's a great time to discuss should we make Scala 2.12 
as default Scala version in Spark 3.0.

Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely to 
support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed work 
per discussion in Scala community, 
https://github.com/scala/scala-dev/issues/559#issuecomment-436160166

We have initial support of Scala 2.12 in Spark 2.4. If we decide to make Scala 
2.12 as default for Spark 3.0 now, we will have ample time to work on bugs and 
issues that we may run into.

What do you think?

Thanks,

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc


-
To unsubscribe e-mail: 
dev-unsubscr...@spark.apache.org




--
Ryan Blue
Software Engineer
Netflix

Re: Test and support only LTS JDK release?

[Ryan Blue]

+1 for supporting LTS releases.

On Tue, Nov 6, 2018 at 11:48 AM Robert Stupp  wrote:

> +1 on supporting LTS releases.
>
> VM distributors (RedHat, Azul - to name two) want to provide patches to
> LTS versions (i.e. into http://hg.openjdk.java.net/jdk-updates/jdk11u/).
> How that will play out in reality ... I don't know. Whether Oracle will
> contribute to that repo for 8 after it's EOL and 11 after the 6 month cycle
> ... we will see. Most Linux distributions promised(?) long-term support for
> Java 11 in their LTS releases (e.g. Ubuntu 18.04). I am not sure what that
> exactly means ... whether they will actively provide patches to OpenJDK or
> whether they just build from source.
>
> But considering that, I think it's definitely worth to at least keep an
> eye on Java 12 and 13 - even if those are just EA. Java 12 for example does
> already forbid some "dirty tricks" that are still possible in Java 11.
>
>
> On 11/6/18 8:32 PM, DB Tsai wrote:
>
> OpenJDK will follow Oracle's release cycle,
> https://openjdk.java.net/projects/jdk/, a strict six months model. I'm
> not familiar with other non-Oracle VMs and Redhat support.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
>
> On Nov 6, 2018, at 11:26 AM, Reynold Xin  wrote:
>
> What does OpenJDK do and other non-Oracle VMs? I know there was a lot of
> discussions from Redhat etc to support.
>
>
> On Tue, Nov 6, 2018 at 11:24 AM DB Tsai  wrote:
>
>> Given Oracle's new 6-month release model, I feel the only realistic
>> option is to only test and support JDK such as JDK 11 LTS and future LTS
>> release. I would like to have a discussion on this in Spark community.
>>
>> Thanks,
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
>> Apple, Inc
>>
>>
> --
> Robert Stupp
> @snazy
>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Make Scala 2.12 as default Scala version in Spark 3.0

[Ryan Blue]

+1 to Scala 2.12 as the default in Spark 3.0.

On Tue, Nov 6, 2018 at 11:50 AM DB Tsai  wrote:

> +1 on dropping Scala 2.11 in Spark 3.0 to simplify the build.
>
> As Scala 2.11 will not support Java 11 unless we make a significant
> investment, if we decide not to drop Scala 2.11 in Spark 3.0, what we can
> do is have only Scala 2.12 build support Java 11 while Scala 2.11 support
> Java 8. But I agree with Sean that this can make the decencies really
> complicated; hence I support to drop Scala 2.11 in Spark 3.0 directly.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
>
> On Nov 6, 2018, at 11:38 AM, Sean Owen  wrote:
>
> I think we should make Scala 2.12 the default in Spark 3.0. I would
> also prefer to drop Scala 2.11 support in 3.0. In theory, not dropping
> 2.11 support it means we'd support Scala 2.11 for years, the lifetime
> of Spark 3.x. In practice, we could drop 2.11 support in a 3.1.0 or
> 3.2.0 release, kind of like what happened with 2.10 in 2.x.
>
> Java (9-)11 support also complicates this. I think getting it to work
> will need some significant dependency updates, and I worry not all
> will be available for 2.11 or will present some knotty problems. We'll
> find out soon if that forces the issue.
>
> Also note that Scala 2.13 is pretty close to release, and we'll want
> to support it soon after release, perhaps sooner than the long delay
> before 2.12 was supported (because it was hard!). It will probably be
> out well before Spark 3.0. Cross-compiling for 3 Scala versions sounds
> like too much. 3.0 could support 2.11 and 2.12, and 3.1 support 2.12
> and 2.13, or something. But if 2.13 support is otherwise attainable at
> the release of Spark 3.0, I wonder if that too argues for dropping
> 2.11 support.
>
> Finally I'll say that Spark itself isn't dropping 2.11 support for a
> while, no matter what; it still exists in the 2.4.x branch of course.
> People who can't update off Scala 2.11 can stay on Spark 2.x, note.
>
> Sean
>
>
> On Tue, Nov 6, 2018 at 1:13 PM DB Tsai  wrote:
>
>
> We made Scala 2.11 as default Scala version in Spark 2.0. Now, the next
> Spark version will be 3.0, so it's a great time to discuss should we make
> Scala 2.12 as default Scala version in Spark 3.0.
>
> Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely to
> support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed
> work per discussion in Scala community,
> https://github.com/scala/scala-dev/issues/559#issuecomment-436160166
>
> We have initial support of Scala 2.12 in Spark 2.4. If we decide to make
> Scala 2.12 as default for Spark 3.0 now, we will have ample time to work on
> bugs and issues that we may run into.
>
> What do you think?
>
> Thanks,
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> 
>
>
>

-- 
Ryan Blue
Software Engineer
Netflix

Re: Make Scala 2.12 as default Scala version in Spark 3.0

[DB Tsai]

+1 on dropping Scala 2.11 in Spark 3.0 to simplify the build. 

As Scala 2.11 will not support Java 11 unless we make a significant investment, 
if we decide not to drop Scala 2.11 in Spark 3.0, what we can do is have only 
Scala 2.12 build support Java 11 while Scala 2.11 support Java 8. But I agree 
with Sean that this can make the decencies really complicated; hence I support 
to drop Scala 2.11 in Spark 3.0 directly.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Nov 6, 2018, at 11:38 AM, Sean Owen  wrote:
> 
> I think we should make Scala 2.12 the default in Spark 3.0. I would
> also prefer to drop Scala 2.11 support in 3.0. In theory, not dropping
> 2.11 support it means we'd support Scala 2.11 for years, the lifetime
> of Spark 3.x. In practice, we could drop 2.11 support in a 3.1.0 or
> 3.2.0 release, kind of like what happened with 2.10 in 2.x.
> 
> Java (9-)11 support also complicates this. I think getting it to work
> will need some significant dependency updates, and I worry not all
> will be available for 2.11 or will present some knotty problems. We'll
> find out soon if that forces the issue.
> 
> Also note that Scala 2.13 is pretty close to release, and we'll want
> to support it soon after release, perhaps sooner than the long delay
> before 2.12 was supported (because it was hard!). It will probably be
> out well before Spark 3.0. Cross-compiling for 3 Scala versions sounds
> like too much. 3.0 could support 2.11 and 2.12, and 3.1 support 2.12
> and 2.13, or something. But if 2.13 support is otherwise attainable at
> the release of Spark 3.0, I wonder if that too argues for dropping
> 2.11 support.
> 
> Finally I'll say that Spark itself isn't dropping 2.11 support for a
> while, no matter what; it still exists in the 2.4.x branch of course.
> People who can't update off Scala 2.11 can stay on Spark 2.x, note.
> 
> Sean
> 
> 
> On Tue, Nov 6, 2018 at 1:13 PM DB Tsai  wrote:
>> 
>> We made Scala 2.11 as default Scala version in Spark 2.0. Now, the next 
>> Spark version will be 3.0, so it's a great time to discuss should we make 
>> Scala 2.12 as default Scala version in Spark 3.0.
>> 
>> Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely to 
>> support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed work 
>> per discussion in Scala community, 
>> https://github.com/scala/scala-dev/issues/559#issuecomment-436160166
>> 
>> We have initial support of Scala 2.12 in Spark 2.4. If we decide to make 
>> Scala 2.12 as default for Spark 3.0 now, we will have ample time to work on 
>> bugs and issues that we may run into.
>> 
>> What do you think?
>> 
>> Thanks,
>> 
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
>> Inc
>> 
>> 
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> 

Re: Test and support only LTS JDK release?

[Robert Stupp]

+1 on supporting LTS releases.

VM distributors (RedHat, Azul - to name two) want to provide patches to 
LTS versions (i.e. into http://hg.openjdk.java.net/jdk-updates/jdk11u/). 
How that will play out in reality ... I don't know. Whether Oracle will 
contribute to that repo for 8 after it's EOL and 11 after the 6 month 
cycle ... we will see. Most Linux distributions promised(?) long-term 
support for Java 11 in their LTS releases (e.g. Ubuntu 18.04). I am not 
sure what that exactly means ... whether they will actively provide 
patches to OpenJDK or whether they just build from source.


But considering that, I think it's definitely worth to at least keep an 
eye on Java 12 and 13 - even if those are just EA. Java 12 for example 
does already forbid some "dirty tricks" that are still possible in Java 11.



On 11/6/18 8:32 PM, DB Tsai wrote:
OpenJDK will follow Oracle's release cycle, 
https://openjdk.java.net/projects/jdk/, a strict six months model. I'm 
not familiar with other non-Oracle VMs and Redhat support.


DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   
Apple, Inc


On Nov 6, 2018, at 11:26 AM, Reynold Xin > wrote:


What does OpenJDK do and other non-Oracle VMs? I know there was a lot 
of discussions from Redhat etc to support.



On Tue, Nov 6, 2018 at 11:24 AM DB Tsai > wrote:


Given Oracle's new 6-month release model, I feel the only
realistic option is to only test and support JDK such as JDK 11
LTS and future LTS release. I would like to have a discussion on
this in Spark community.

Thanks,

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |
  Apple, Inc




--
Robert Stupp
@snazy

Re: Make Scala 2.12 as default Scala version in Spark 3.0

[Sean Owen]

I think we should make Scala 2.12 the default in Spark 3.0. I would
also prefer to drop Scala 2.11 support in 3.0. In theory, not dropping
2.11 support it means we'd support Scala 2.11 for years, the lifetime
of Spark 3.x. In practice, we could drop 2.11 support in a 3.1.0 or
3.2.0 release, kind of like what happened with 2.10 in 2.x.

Java (9-)11 support also complicates this. I think getting it to work
will need some significant dependency updates, and I worry not all
will be available for 2.11 or will present some knotty problems. We'll
find out soon if that forces the issue.

Also note that Scala 2.13 is pretty close to release, and we'll want
to support it soon after release, perhaps sooner than the long delay
before 2.12 was supported (because it was hard!). It will probably be
out well before Spark 3.0. Cross-compiling for 3 Scala versions sounds
like too much. 3.0 could support 2.11 and 2.12, and 3.1 support 2.12
and 2.13, or something. But if 2.13 support is otherwise attainable at
the release of Spark 3.0, I wonder if that too argues for dropping
2.11 support.

Finally I'll say that Spark itself isn't dropping 2.11 support for a
while, no matter what; it still exists in the 2.4.x branch of course.
People who can't update off Scala 2.11 can stay on Spark 2.x, note.

Sean


On Tue, Nov 6, 2018 at 1:13 PM DB Tsai  wrote:
>
> We made Scala 2.11 as default Scala version in Spark 2.0. Now, the next Spark 
> version will be 3.0, so it's a great time to discuss should we make Scala 
> 2.12 as default Scala version in Spark 3.0.
>
> Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely to 
> support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed work 
> per discussion in Scala community, 
> https://github.com/scala/scala-dev/issues/559#issuecomment-436160166
>
> We have initial support of Scala 2.12 in Spark 2.4. If we decide to make 
> Scala 2.12 as default for Spark 3.0 now, we will have ample time to work on 
> bugs and issues that we may run into.
>
> What do you think?
>
> Thanks,
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
> Inc
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Test and support only LTS JDK release?

[DB Tsai]

OpenJDK will follow Oracle's release cycle, 
https://openjdk.java.net/projects/jdk/ 
, a strict six months model. I'm not 
familiar with other non-Oracle VMs and Redhat support.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Nov 6, 2018, at 11:26 AM, Reynold Xin  wrote:
> 
> What does OpenJDK do and other non-Oracle VMs? I know there was a lot of 
> discussions from Redhat etc to support.
> 
> 
> On Tue, Nov 6, 2018 at 11:24 AM DB Tsai  > wrote:
> Given Oracle's new 6-month release model, I feel the only realistic option is 
> to only test and support JDK such as JDK 11 LTS and future LTS release. I 
> would like to have a discussion on this in Spark community.  
> 
> Thanks,
> 
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
> Inc
> 

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

[Felix Cheung]

I’d rather not mess with 2.4.0 at this point. On CRAN is nice but users can 
also install from Apache Mirror.

Also I had attempted and failed to get vignettes not to build, it was non 
trivial and could t get it to work.  It I have an idea.

As for tests I don’t know exact why is it not skipped. Need to investigate but 
worse case test_package can run with 0 test.




From: Sean Owen 
Sent: Tuesday, November 6, 2018 10:51 AM
To: Shivaram Venkataraman
Cc: Felix Cheung; Wenchen Fan; Matei Zaharia; dev
Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

I think the second option, to skip the tests, is best right now, if
the alternative is to have no SparkR release at all!
Can we monkey-patch the 2.4.0 release for SparkR in this way, bless it
from the PMC, and release that? It's drastic but so is not being able
to release, I think.
Right? or is CRAN not actually an important distribution path for
SparkR in particular?

On Tue, Nov 6, 2018 at 12:49 PM Shivaram Venkataraman
 wrote:
>
> Right - I think we should move on with 2.4.0.
>
> In terms of what can be done to avoid this error there are two strategies
> - Felix had this other thread about JDK 11 that should at least let
> Spark run on the CRAN instance. In general this strategy isn't
> foolproof because the JDK version and other dependencies on that
> machine keep changing over time and we dont have much control over it.
> Worse we also dont have much control
> - The other solution is to not run code to build the vignettes
> document and just have static code blocks there that have been
> pre-evaluated / pre-populated. We can open a JIRA to discuss the
> pros/cons of this ?
>
> Thanks
> Shivaram
>
> On Tue, Nov 6, 2018 at 10:57 AM Felix Cheung  
> wrote:
> >
> > We have not been able to publish to CRAN for quite some time (since 2.3.0 
> > was archived - the cause is Java 11)
> >
> > I think it’s ok to announce the release of 2.4.0
> >
> >
> > 
> > From: Wenchen Fan 
> > Sent: Tuesday, November 6, 2018 8:51 AM
> > To: Felix Cheung
> > Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > Do you mean we should have a 2.4.0 release without CRAN and then do a 2.4.1 
> > immediately?
> >
> > On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung  
> > wrote:
> >>
> >> Shivaram and I were discussing.
> >> Actually we worked with them before. Another possible approach is to 
> >> remove the vignettes eval and all test from the source package... in the 
> >> next release.
> >>
> >>
> >> 
> >> From: Matei Zaharia 
> >> Sent: Tuesday, November 6, 2018 12:07 AM
> >> To: Felix Cheung
> >> Cc: Sean Owen; dev; Shivaram Venkataraman
> >> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >>
> >> Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps 
> >> we aren’t disabling it correctly, or perhaps they can ignore this specific 
> >> failure. +Shivaram who might have some ideas.
> >>
> >> Matei
> >>
> >> > On Nov 5, 2018, at 9:09 PM, Felix Cheung  
> >> > wrote:
> >> >
> >> > I don¡Št know what the cause is yet.
> >> >
> >> > The test should be skipped because of this check
> >> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
> >> >
> >> > And this
> >> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
> >> >
> >> > But it ran:
> >> > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> >> > "fit", formula,
> >> >
> >> > The earlier release was achived because of Java 11+ too so this 
> >> > unfortunately isn¡Št new.
> >> >
> >> >
> >> > From: Sean Owen 
> >> > Sent: Monday, November 5, 2018 7:22 PM
> >> > To: Felix Cheung
> >> > Cc: dev
> >> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >> >
> >> > What can we do to get the release through? is there any way to
> >> > circumvent these tests or otherwise hack it? or does it need a
> >> > maintenance release?
> >> > On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung  
> >> > wrote:
> >> > >
> >> > > FYI. SparkR submission failed. It seems to detect Java 11 correctly 
> >> > > with vignettes but not skipping tests as would be expected.
> >> > >
> >> > > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with 
> >> > > diagnostics:
> >> > > Java version 8 is required for this package; found version: 11.0.1
> >> > > Execution halted
> >> > >
> >> > > * checking PDF version of manual ... OK
> >> > > * DONE
> >> > > Status: 1 WARNING, 1 NOTE
> >> > >
> >> > > Current CRAN status: ERROR: 1, OK: 1
> >> > > See: 
> >> > >
> >> > > Version: 2.3.0
> >> > > Check: tests, Result: ERROR
> >> > > Running ¡¥run-all.R¡Š [8s/35s]
> >> > > Running the tests in ¡¥tests/run-all.R¡Š failed.
> >> > > Last 13 lines of 

Re: Test and support only LTS JDK release?

[Marcelo Vanzin]

+1, that's always been my view.

Although, to be fair, and as Sean mentioned, the jump from jdk8 is
probably the harder part. After that it's less likely (hopefully?)
that we'll run into issues in non-LTS releases. And even if we don't
officially support them, trying to keep up with breaking changes might
make it easier to support the following LTS.

(Just as an example, both jdk9 and jdk10 are already EOL.)

On Tue, Nov 6, 2018 at 11:24 AM DB Tsai  wrote:
>
> Given Oracle's new 6-month release model, I feel the only realistic option is 
> to only test and support JDK such as JDK 11 LTS and future LTS release. I 
> would like to have a discussion on this in Spark community.
>
> Thanks,
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
> Inc
>


-- 
Marcelo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Test and support only LTS JDK release?

[Reynold Xin]

What does OpenJDK do and other non-Oracle VMs? I know there was a lot of
discussions from Redhat etc to support.


On Tue, Nov 6, 2018 at 11:24 AM DB Tsai  wrote:

> Given Oracle's new 6-month release model, I feel the only realistic option
> is to only test and support JDK such as JDK 11 LTS and future LTS release.
> I would like to have a discussion on this in Spark community.
>
> Thanks,
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
>
>

Test and support only LTS JDK release?

[DB Tsai]

Given Oracle's new 6-month release model, I feel the only realistic option is 
to only test and support JDK such as JDK 11 LTS and future LTS release. I would 
like to have a discussion on this in Spark community.  

Thanks,

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

Re: Make Scala 2.12 as default Scala version in Spark 3.0

[Dongjoon Hyun]

+1 for making Scala 2.12 as default for Spark 3.0.

Bests,
Dongjoon.


On Tue, Nov 6, 2018 at 11:13 AM DB Tsai  wrote:

> We made Scala 2.11 as default Scala version in Spark 2.0. Now, the next
> Spark version will be 3.0, so it's a great time to discuss should we make
> Scala 2.12 as default Scala version in Spark 3.0.
>
> Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely to
> support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed
> work per discussion in Scala community,
> https://github.com/scala/scala-dev/issues/559#issuecomment-436160166
>
> We have initial support of Scala 2.12 in Spark 2.4. If we decide to make
> Scala 2.12 as default for Spark 3.0 now, we will have ample time to work on
> bugs and issues that we may run into.
>
> What do you think?
>
> Thanks,
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Make Scala 2.12 as default Scala version in Spark 3.0

[DB Tsai]

We made Scala 2.11 as default Scala version in Spark 2.0. Now, the next Spark 
version will be 3.0, so it's a great time to discuss should we make Scala 2.12 
as default Scala version in Spark 3.0.

Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely to 
support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed work 
per discussion in Scala community, 
https://github.com/scala/scala-dev/issues/559#issuecomment-436160166

We have initial support of Scala 2.12 in Spark 2.4. If we decide to make Scala 
2.12 as default for Spark 3.0 now, we will have ample time to work on bugs and 
issues that we may run into.

What do you think?

Thanks, 

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Java 11 support

[DB Tsai]

Scala 2.11 is EOL, and only Scala 2.12 will support JDK 11 
https://github.com/scala/scala-dev/issues/559#issuecomment-436160166 
 , we 
might need to make Scala 2.12 as default version in Spark 3.0 to move forward. 

Given Oracle's new 6-month release model, I think the only realistic option is 
to only support and test LTS JDK. I'll send out two separate emails to dev to 
facilitate the discussion. 

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Nov 6, 2018, at 9:47 AM, shane knapp  wrote:
> 
> cool, i was wondering when we were going to forge ahead in to the great 
> future of jdk8++...  i went ahead and created a sub-task of installing a 
> newer version of java on the build nodes 
> (https://issues.apache.org/jira/browse/SPARK-25953 
> ), and once we figure out 
> exact what version we want i'll go ahead and get that done.
> 
> On Tue, Nov 6, 2018 at 9:11 AM Sean Owen  > wrote:
> I think that Java 9 support basically gets Java 10, 11 support. But
> the jump from 8 to 9 is unfortunately more breaking than usual because
> of the total revamping of the internal JDK classes. I think it will be
> mostly a matter of dependencies needing updates to work. I agree this
> is probably pretty important for Spark 3. Here's the ticket I know of:
> https://issues.apache.org/jira/browse/SPARK-24417 
>  . DB is already
> working on some of it, I see.
> On Tue, Nov 6, 2018 at 10:59 AM Felix Cheung  > wrote:
> >
> > Speaking of, get we work to support Java 11?
> > That will fix all the problems below.
> >
> >
> >
> > 
> > From: Felix Cheung  > >
> > Sent: Tuesday, November 6, 2018 8:57 AM
> > To: Wenchen Fan
> > Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > We have not been able to publish to CRAN for quite some time (since 2.3.0 
> > was archived - the cause is Java 11)
> >
> > I think it’s ok to announce the release of 2.4.0
> >
> >
> > 
> > From: Wenchen Fan mailto:cloud0...@gmail.com>>
> > Sent: Tuesday, November 6, 2018 8:51 AM
> > To: Felix Cheung
> > Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > Do you mean we should have a 2.4.0 release without CRAN and then do a 2.4.1 
> > immediately?
> >
> > On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung  > > wrote:
> >>
> >> Shivaram and I were discussing.
> >> Actually we worked with them before. Another possible approach is to 
> >> remove the vignettes eval and all test from the source package... in the 
> >> next release.
> >>
> >>
> >> 
> >> From: Matei Zaharia  >> >
> >> Sent: Tuesday, November 6, 2018 12:07 AM
> >> To: Felix Cheung
> >> Cc: Sean Owen; dev; Shivaram Venkataraman
> >> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >>
> >> Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps 
> >> we aren’t disabling it correctly, or perhaps they can ignore this specific 
> >> failure. +Shivaram who might have some ideas.
> >>
> >> Matei
> >>
> >> > On Nov 5, 2018, at 9:09 PM, Felix Cheung  >> > > wrote:
> >> >
> >> > I don¡Št know what the cause is yet.
> >> >
> >> > The test should be skipped because of this check
> >> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
> >> >  
> >> > 
> >> >
> >> > And this
> >> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
> >> >  
> >> > 
> >> >
> >> > But it ran:
> >> > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> >> > "fit", formula,
> >> >
> >> > The earlier release was achived because of Java 11+ too so this 
> >> > unfortunately isn¡Št new.
> >> >
> >> >
> >> > From: Sean Owen mailto:sro...@gmail.com>>
> >> > Sent: Monday, November 5, 2018 7:22 PM
> >> > To: Felix Cheung
> >> > Cc: dev
> >> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >> >
> >> > What can we do to get the release through? is there any way to
> >> > circumvent these tests or otherwise hack it? or does it need a
> >> > maintenance release?
> >> > On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung  >> > > wrote:
> >> > >
> >> > > FYI. SparkR submission failed. It seems to detect Java 11 

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

[Sean Owen]

I think the second option, to skip the tests, is best right now, if
the alternative is to have no SparkR release at all!
Can we monkey-patch the 2.4.0 release for SparkR in this way, bless it
from the PMC, and release that? It's drastic but so is not being able
to release, I think.
Right? or is CRAN not actually an important distribution path for
SparkR in particular?

On Tue, Nov 6, 2018 at 12:49 PM Shivaram Venkataraman
 wrote:
>
> Right - I think we should move on with 2.4.0.
>
> In terms of what can be done to avoid this error there are two strategies
> - Felix had this other thread about JDK 11 that should at least let
> Spark run on the CRAN instance. In general this strategy isn't
> foolproof because the JDK version and other dependencies on that
> machine keep changing over time and we dont have much control over it.
> Worse we also dont have much control
> - The other solution is to not run code to build the vignettes
> document and just have static code blocks there that have been
> pre-evaluated / pre-populated. We can open a JIRA to discuss the
> pros/cons of this  ?
>
> Thanks
> Shivaram
>
> On Tue, Nov 6, 2018 at 10:57 AM Felix Cheung  
> wrote:
> >
> > We have not been able to publish to CRAN for quite some time (since 2.3.0 
> > was archived - the cause is Java 11)
> >
> > I think it’s ok to announce the release of 2.4.0
> >
> >
> > 
> > From: Wenchen Fan 
> > Sent: Tuesday, November 6, 2018 8:51 AM
> > To: Felix Cheung
> > Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > Do you mean we should have a 2.4.0 release without CRAN and then do a 2.4.1 
> > immediately?
> >
> > On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung  
> > wrote:
> >>
> >> Shivaram and I were discussing.
> >> Actually we worked with them before. Another possible approach is to 
> >> remove the vignettes eval and all test from the source package... in the 
> >> next release.
> >>
> >>
> >> 
> >> From: Matei Zaharia 
> >> Sent: Tuesday, November 6, 2018 12:07 AM
> >> To: Felix Cheung
> >> Cc: Sean Owen; dev; Shivaram Venkataraman
> >> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >>
> >> Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps 
> >> we aren’t disabling it correctly, or perhaps they can ignore this specific 
> >> failure. +Shivaram who might have some ideas.
> >>
> >> Matei
> >>
> >> > On Nov 5, 2018, at 9:09 PM, Felix Cheung  
> >> > wrote:
> >> >
> >> > I don¡Št know what the cause is yet.
> >> >
> >> > The test should be skipped because of this check
> >> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
> >> >
> >> > And this
> >> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
> >> >
> >> > But it ran:
> >> > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> >> > "fit", formula,
> >> >
> >> > The earlier release was achived because of Java 11+ too so this 
> >> > unfortunately isn¡Št new.
> >> >
> >> >
> >> > From: Sean Owen 
> >> > Sent: Monday, November 5, 2018 7:22 PM
> >> > To: Felix Cheung
> >> > Cc: dev
> >> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >> >
> >> > What can we do to get the release through? is there any way to
> >> > circumvent these tests or otherwise hack it? or does it need a
> >> > maintenance release?
> >> > On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung  
> >> > wrote:
> >> > >
> >> > > FYI. SparkR submission failed. It seems to detect Java 11 correctly 
> >> > > with vignettes but not skipping tests as would be expected.
> >> > >
> >> > > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with 
> >> > > diagnostics:
> >> > > Java version 8 is required for this package; found version: 11.0.1
> >> > > Execution halted
> >> > >
> >> > > * checking PDF version of manual ... OK
> >> > > * DONE
> >> > > Status: 1 WARNING, 1 NOTE
> >> > >
> >> > > Current CRAN status: ERROR: 1, OK: 1
> >> > > See: 
> >> > >
> >> > > Version: 2.3.0
> >> > > Check: tests, Result: ERROR
> >> > > Running ¡¥run-all.R¡Š [8s/35s]
> >> > > Running the tests in ¡¥tests/run-all.R¡Š failed.
> >> > > Last 13 lines of output:
> >> > > 4: 
> >> > > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper",
> >> > >  "fit", formula,
> >> > > data@sdf, tolower(family$family), family$link, tol, 
> >> > > as.integer(maxIter), weightCol,
> >> > > regParam, as.double(var.power), as.double(link.power), 
> >> > > stringIndexerOrderType,
> >> > > offsetCol)
> >> > > 5: invokeJava(isStatic = TRUE, className, methodName, ...)
> >> > > 6: handleErrors(returnStatus, conn)
> >> > > 7: stop(readString(conn))
> >> > >
> >> > >  testthat results 
> >> > > 

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

[Shivaram Venkataraman]

Right - I think we should move on with 2.4.0.

In terms of what can be done to avoid this error there are two strategies
- Felix had this other thread about JDK 11 that should at least let
Spark run on the CRAN instance. In general this strategy isn't
foolproof because the JDK version and other dependencies on that
machine keep changing over time and we dont have much control over it.
Worse we also dont have much control
- The other solution is to not run code to build the vignettes
document and just have static code blocks there that have been
pre-evaluated / pre-populated. We can open a JIRA to discuss the
pros/cons of this  ?

Thanks
Shivaram

On Tue, Nov 6, 2018 at 10:57 AM Felix Cheung  wrote:
>
> We have not been able to publish to CRAN for quite some time (since 2.3.0 was 
> archived - the cause is Java 11)
>
> I think it’s ok to announce the release of 2.4.0
>
>
> 
> From: Wenchen Fan 
> Sent: Tuesday, November 6, 2018 8:51 AM
> To: Felix Cheung
> Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>
> Do you mean we should have a 2.4.0 release without CRAN and then do a 2.4.1 
> immediately?
>
> On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung  
> wrote:
>>
>> Shivaram and I were discussing.
>> Actually we worked with them before. Another possible approach is to remove 
>> the vignettes eval and all test from the source package... in the next 
>> release.
>>
>>
>> 
>> From: Matei Zaharia 
>> Sent: Tuesday, November 6, 2018 12:07 AM
>> To: Felix Cheung
>> Cc: Sean Owen; dev; Shivaram Venkataraman
>> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>>
>> Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps we 
>> aren’t disabling it correctly, or perhaps they can ignore this specific 
>> failure. +Shivaram who might have some ideas.
>>
>> Matei
>>
>> > On Nov 5, 2018, at 9:09 PM, Felix Cheung  wrote:
>> >
>> > I don¡Št know what the cause is yet.
>> >
>> > The test should be skipped because of this check
>> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
>> >
>> > And this
>> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
>> >
>> > But it ran:
>> > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
>> > "fit", formula,
>> >
>> > The earlier release was achived because of Java 11+ too so this 
>> > unfortunately isn¡Št new.
>> >
>> >
>> > From: Sean Owen 
>> > Sent: Monday, November 5, 2018 7:22 PM
>> > To: Felix Cheung
>> > Cc: dev
>> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>> >
>> > What can we do to get the release through? is there any way to
>> > circumvent these tests or otherwise hack it? or does it need a
>> > maintenance release?
>> > On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung  
>> > wrote:
>> > >
>> > > FYI. SparkR submission failed. It seems to detect Java 11 correctly with 
>> > > vignettes but not skipping tests as would be expected.
>> > >
>> > > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with 
>> > > diagnostics:
>> > > Java version 8 is required for this package; found version: 11.0.1
>> > > Execution halted
>> > >
>> > > * checking PDF version of manual ... OK
>> > > * DONE
>> > > Status: 1 WARNING, 1 NOTE
>> > >
>> > > Current CRAN status: ERROR: 1, OK: 1
>> > > See: 
>> > >
>> > > Version: 2.3.0
>> > > Check: tests, Result: ERROR
>> > > Running ¡¥run-all.R¡Š [8s/35s]
>> > > Running the tests in ¡¥tests/run-all.R¡Š failed.
>> > > Last 13 lines of output:
>> > > 4: 
>> > > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
>> > > "fit", formula,
>> > > data@sdf, tolower(family$family), family$link, tol, as.integer(maxIter), 
>> > > weightCol,
>> > > regParam, as.double(var.power), as.double(link.power), 
>> > > stringIndexerOrderType,
>> > > offsetCol)
>> > > 5: invokeJava(isStatic = TRUE, className, methodName, ...)
>> > > 6: handleErrors(returnStatus, conn)
>> > > 7: stop(readString(conn))
>> > >
>> > >  testthat results 
>> > > ùù
>> > > OK: 0 SKIPPED: 0 FAILED: 2
>> > > 1. Error: create DataFrame from list or data.frame (@test_basic.R#26)
>> > > 2. Error: spark.glm and predict (@test_basic.R#58)
>> > >
>> > >
>> > >
>> > > -- Forwarded message -
>> > > Date: Mon, Nov 5, 2018, 10:12
>> > > Subject: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>> > >
>> > > Dear maintainer,
>> > >
>> > > package SparkR_2.4.0.tar.gz does not pass the incoming checks 
>> > > automatically, please see the following pre-tests:
>> > > Windows: 
>> > > 

Re: Java 11 support

[shane knapp]

cool, i was wondering when we were going to forge ahead in to the great
future of jdk8++...  i went ahead and created a sub-task of installing a
newer version of java on the build nodes (
https://issues.apache.org/jira/browse/SPARK-25953), and once we figure out
exact what version we want i'll go ahead and get that done.

On Tue, Nov 6, 2018 at 9:11 AM Sean Owen  wrote:

> I think that Java 9 support basically gets Java 10, 11 support. But
> the jump from 8 to 9 is unfortunately more breaking than usual because
> of the total revamping of the internal JDK classes. I think it will be
> mostly a matter of dependencies needing updates to work. I agree this
> is probably pretty important for Spark 3. Here's the ticket I know of:
> https://issues.apache.org/jira/browse/SPARK-24417 . DB is already
> working on some of it, I see.
> On Tue, Nov 6, 2018 at 10:59 AM Felix Cheung 
> wrote:
> >
> > Speaking of, get we work to support Java 11?
> > That will fix all the problems below.
> >
> >
> >
> > 
> > From: Felix Cheung 
> > Sent: Tuesday, November 6, 2018 8:57 AM
> > To: Wenchen Fan
> > Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > We have not been able to publish to CRAN for quite some time (since
> 2.3.0 was archived - the cause is Java 11)
> >
> > I think it’s ok to announce the release of 2.4.0
> >
> >
> > 
> > From: Wenchen Fan 
> > Sent: Tuesday, November 6, 2018 8:51 AM
> > To: Felix Cheung
> > Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > Do you mean we should have a 2.4.0 release without CRAN and then do a
> 2.4.1 immediately?
> >
> > On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung 
> wrote:
> >>
> >> Shivaram and I were discussing.
> >> Actually we worked with them before. Another possible approach is to
> remove the vignettes eval and all test from the source package... in the
> next release.
> >>
> >>
> >> 
> >> From: Matei Zaharia 
> >> Sent: Tuesday, November 6, 2018 12:07 AM
> >> To: Felix Cheung
> >> Cc: Sean Owen; dev; Shivaram Venkataraman
> >> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >>
> >> Maybe it’s wroth contacting the CRAN maintainers to ask for help?
> Perhaps we aren’t disabling it correctly, or perhaps they can ignore this
> specific failure. +Shivaram who might have some ideas.
> >>
> >> Matei
> >>
> >> > On Nov 5, 2018, at 9:09 PM, Felix Cheung 
> wrote:
> >> >
> >> > I don¡Št know what the cause is yet.
> >> >
> >> > The test should be skipped because of this check
> >> >
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
> >> >
> >> > And this
> >> >
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
> >> >
> >> > But it ran:
> >> >
> callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper",
> "fit", formula,
> >> >
> >> > The earlier release was achived because of Java 11+ too so this
> unfortunately isn¡Št new.
> >> >
> >> >
> >> > From: Sean Owen 
> >> > Sent: Monday, November 5, 2018 7:22 PM
> >> > To: Felix Cheung
> >> > Cc: dev
> >> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >> >
> >> > What can we do to get the release through? is there any way to
> >> > circumvent these tests or otherwise hack it? or does it need a
> >> > maintenance release?
> >> > On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung <
> felixcheun...@hotmail.com> wrote:
> >> > >
> >> > > FYI. SparkR submission failed. It seems to detect Java 11 correctly
> with vignettes but not skipping tests as would be expected.
> >> > >
> >> > > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with
> diagnostics:
> >> > > Java version 8 is required for this package; found version: 11.0.1
> >> > > Execution halted
> >> > >
> >> > > * checking PDF version of manual ... OK
> >> > > * DONE
> >> > > Status: 1 WARNING, 1 NOTE
> >> > >
> >> > > Current CRAN status: ERROR: 1, OK: 1
> >> > > See: <
> https://CRAN.R-project.org/web/checks/check_results_SparkR.html>
> >> > >
> >> > > Version: 2.3.0
> >> > > Check: tests, Result: ERROR
> >> > > Running ¡¥run-all.R¡Š [8s/35s]
> >> > > Running the tests in ¡¥tests/run-all.R¡Š failed.
> >> > > Last 13 lines of output:
> >> > > 4:
> callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper",
> "fit", formula,
> >> > > data@sdf, tolower(family$family), family$link, tol,
> as.integer(maxIter), weightCol,
> >> > > regParam, as.double(var.power), as.double(link.power),
> stringIndexerOrderType,
> >> > > offsetCol)
> >> > > 5: invokeJava(isStatic = TRUE, className, methodName, ...)
> >> > > 6: handleErrors(returnStatus, conn)
> >> > > 7: stop(readString(conn))
> >> > >
> >> > >  testthat results
> 

Re: need assistance debugging a strange build failure...

[shane knapp]

btw, this is a compilation error in the SBT build that only shows up on the
ubuntu workers.

On Mon, Nov 5, 2018 at 5:07 PM shane knapp  wrote:

> the maven build is quite happy:
>
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7-ubuntu-testing/
>
> i've wiped all the cached deps from .ivy2 on one ubuntu worker, pinned the
> build to that (research-jenkins-worker-08) and re-launched the sbt build.
> let's see if it fails in a new, spectacular way (or just the same as
> before).
>
> but again:  why is this failing on the ubuntu nodes, vs centos?  if it's a
> poisoned cache, i will be sad, but that's easy to fix.  if it's a bad dep
> in the pom, then...  ¯\_(ツ)_/¯
>
> On Mon, Nov 5, 2018 at 5:00 PM Wenchen Fan  wrote:
>
>> Have you tried Maven instead of SBT? This looks like a Java dependency
>> problem, e.g. a wrong version of Avro is picked.
>>
>> On Tue, Nov 6, 2018 at 8:30 AM shane knapp  wrote:
>>
>>> i'm really close (for real: really close!) on the ubuntu port...  but
>>> one build has been a thorn in my side and i was wondering if i could get
>>> some extra eyes on this as i grind through the remaining few pieces of my
>>> own personal system dependency hell.  :)
>>>
>>> the job in question is:
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-ubuntu-testing
>>>
>>> it's identical to the regular spark-master-test-sbt-hadoop-2.7 job,
>>> except i'm building against a newer version of java (1.8.0_171 vs
>>> 1.8.0_60).
>>>
>>> the centos job always passes on every worker.
>>>
>>> the ubuntu job fails on every ubuntu worker during the scala unidoc
>>> generation w/the following error:
>>>
>>> """
>>> [error]
>>> /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-ubuntu-testing/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123:
>>> value createDatumWriter is not a member of
>>> org.apache.avro.generic.GenericData
>>> [error] writerCache.getOrElseUpdate(schema,
>>> GenericData.get.createDatumWriter(schema))
>>> [error] ^
>>> [info] No documentation generated with unsuccessful compiler run
>>> [error] one error found
>>>
>>> """
>>> an example job w/this failure is here:
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-ubuntu-testing/30/consoleFull
>>>
>>> thoughts?  am i missing something obvious?  i've checked and there are
>>> no avro system packages installed on any of the workers (centos or ubuntu).
>>>
>>> thanks in advance,
>>>
>>> shane
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>


-- 
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: Removing non-deprecated R methods that were deprecated in Python, Scala?

[Shivaram Venkataraman]

Yep. That sounds good to me.
On Tue, Nov 6, 2018 at 11:06 AM Sean Owen  wrote:
>
> Sounds good, remove in 3.1? I can update accordingly.
>
> On Tue, Nov 6, 2018, 10:46 AM Reynold Xin >
>> Maybe deprecate and remove in next version? It is bad to just remove a 
>> method without deprecation notice.
>>
>> On Tue, Nov 6, 2018 at 5:44 AM Sean Owen  wrote:
>>>
>>> See https://github.com/apache/spark/pull/22921#discussion_r230568058
>>>
>>> Methods like toDegrees, toRadians, approxCountDistinct were 'renamed'
>>> in Spark 2.1: deprecated, and replaced with an identical method with
>>> different name. However, these weren't actually deprecated in SparkR.
>>>
>>> Is it an oversight that we should just correct anyway by removing, to
>>> stay synced?
>>> Or deprecate and retain these in Spark 3.0.0?
>>>
>>> Sean
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Java 11 support

[Felix Cheung]

+1 for Spark 3, definitely
Thanks for the updates



From: Sean Owen 
Sent: Tuesday, November 6, 2018 9:11 AM
To: Felix Cheung
Cc: dev
Subject: Re: Java 11 support

I think that Java 9 support basically gets Java 10, 11 support. But
the jump from 8 to 9 is unfortunately more breaking than usual because
of the total revamping of the internal JDK classes. I think it will be
mostly a matter of dependencies needing updates to work. I agree this
is probably pretty important for Spark 3. Here's the ticket I know of:
https://issues.apache.org/jira/browse/SPARK-24417 . DB is already
working on some of it, I see.
On Tue, Nov 6, 2018 at 10:59 AM Felix Cheung  wrote:
>
> Speaking of, get we work to support Java 11?
> That will fix all the problems below.
>
>
>
> 
> From: Felix Cheung 
> Sent: Tuesday, November 6, 2018 8:57 AM
> To: Wenchen Fan
> Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>
> We have not been able to publish to CRAN for quite some time (since 2.3.0 was 
> archived - the cause is Java 11)
>
> I think it’s ok to announce the release of 2.4.0
>
>
> 
> From: Wenchen Fan 
> Sent: Tuesday, November 6, 2018 8:51 AM
> To: Felix Cheung
> Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>
> Do you mean we should have a 2.4.0 release without CRAN and then do a 2.4.1 
> immediately?
>
> On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung  
> wrote:
>>
>> Shivaram and I were discussing.
>> Actually we worked with them before. Another possible approach is to remove 
>> the vignettes eval and all test from the source package... in the next 
>> release.
>>
>>
>> 
>> From: Matei Zaharia 
>> Sent: Tuesday, November 6, 2018 12:07 AM
>> To: Felix Cheung
>> Cc: Sean Owen; dev; Shivaram Venkataraman
>> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>>
>> Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps we 
>> aren’t disabling it correctly, or perhaps they can ignore this specific 
>> failure. +Shivaram who might have some ideas.
>>
>> Matei
>>
>> > On Nov 5, 2018, at 9:09 PM, Felix Cheung  wrote:
>> >
>> > I don¡Št know what the cause is yet.
>> >
>> > The test should be skipped because of this check
>> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
>> >
>> > And this
>> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
>> >
>> > But it ran:
>> > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
>> > "fit", formula,
>> >
>> > The earlier release was achived because of Java 11+ too so this 
>> > unfortunately isn¡Št new.
>> >
>> >
>> > From: Sean Owen 
>> > Sent: Monday, November 5, 2018 7:22 PM
>> > To: Felix Cheung
>> > Cc: dev
>> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>> >
>> > What can we do to get the release through? is there any way to
>> > circumvent these tests or otherwise hack it? or does it need a
>> > maintenance release?
>> > On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung  
>> > wrote:
>> > >
>> > > FYI. SparkR submission failed. It seems to detect Java 11 correctly with 
>> > > vignettes but not skipping tests as would be expected.
>> > >
>> > > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with 
>> > > diagnostics:
>> > > Java version 8 is required for this package; found version: 11.0.1
>> > > Execution halted
>> > >
>> > > * checking PDF version of manual ... OK
>> > > * DONE
>> > > Status: 1 WARNING, 1 NOTE
>> > >
>> > > Current CRAN status: ERROR: 1, OK: 1
>> > > See: 
>> > >
>> > > Version: 2.3.0
>> > > Check: tests, Result: ERROR
>> > > Running ¡¥run-all.R¡Š [8s/35s]
>> > > Running the tests in ¡¥tests/run-all.R¡Š failed.
>> > > Last 13 lines of output:
>> > > 4: 
>> > > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
>> > > "fit", formula,
>> > > data@sdf, tolower(family$family), family$link, tol, as.integer(maxIter), 
>> > > weightCol,
>> > > regParam, as.double(var.power), as.double(link.power), 
>> > > stringIndexerOrderType,
>> > > offsetCol)
>> > > 5: invokeJava(isStatic = TRUE, className, methodName, ...)
>> > > 6: handleErrors(returnStatus, conn)
>> > > 7: stop(readString(conn))
>> > >
>> > >  testthat results 
>> > > ùù
>> > > OK: 0 SKIPPED: 0 FAILED: 2
>> > > 1. Error: create DataFrame from list or data.frame (@test_basic.R#26)
>> > > 2. Error: spark.glm and predict (@test_basic.R#58)
>> > >
>> > >
>> > >
>> > > -- Forwarded message -
>> > > Date: Mon, Nov 5, 2018, 10:12
>> > > 

Re: Java 11 support

[Sean Owen]

I think that Java 9 support basically gets Java 10, 11 support. But
the jump from 8 to 9 is unfortunately more breaking than usual because
of the total revamping of the internal JDK classes. I think it will be
mostly a matter of dependencies needing updates to work. I agree this
is probably pretty important for Spark 3. Here's the ticket I know of:
https://issues.apache.org/jira/browse/SPARK-24417 . DB is already
working on some of it, I see.
On Tue, Nov 6, 2018 at 10:59 AM Felix Cheung  wrote:
>
> Speaking of, get we work to support Java 11?
> That will fix all the problems below.
>
>
>
> 
> From: Felix Cheung 
> Sent: Tuesday, November 6, 2018 8:57 AM
> To: Wenchen Fan
> Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>
> We have not been able to publish to CRAN for quite some time (since 2.3.0 was 
> archived - the cause is Java 11)
>
> I think it’s ok to announce the release of 2.4.0
>
>
> 
> From: Wenchen Fan 
> Sent: Tuesday, November 6, 2018 8:51 AM
> To: Felix Cheung
> Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>
> Do you mean we should have a 2.4.0 release without CRAN and then do a 2.4.1 
> immediately?
>
> On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung  
> wrote:
>>
>> Shivaram and I were discussing.
>> Actually we worked with them before. Another possible approach is to remove 
>> the vignettes eval and all test from the source package... in the next 
>> release.
>>
>>
>> 
>> From: Matei Zaharia 
>> Sent: Tuesday, November 6, 2018 12:07 AM
>> To: Felix Cheung
>> Cc: Sean Owen; dev; Shivaram Venkataraman
>> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>>
>> Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps we 
>> aren’t disabling it correctly, or perhaps they can ignore this specific 
>> failure. +Shivaram who might have some ideas.
>>
>> Matei
>>
>> > On Nov 5, 2018, at 9:09 PM, Felix Cheung  wrote:
>> >
>> > I don¡Št know what the cause is yet.
>> >
>> > The test should be skipped because of this check
>> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
>> >
>> > And this
>> > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
>> >
>> > But it ran:
>> > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
>> > "fit", formula,
>> >
>> > The earlier release was achived because of Java 11+ too so this 
>> > unfortunately isn¡Št new.
>> >
>> >
>> > From: Sean Owen 
>> > Sent: Monday, November 5, 2018 7:22 PM
>> > To: Felix Cheung
>> > Cc: dev
>> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>> >
>> > What can we do to get the release through? is there any way to
>> > circumvent these tests or otherwise hack it? or does it need a
>> > maintenance release?
>> > On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung  
>> > wrote:
>> > >
>> > > FYI. SparkR submission failed. It seems to detect Java 11 correctly with 
>> > > vignettes but not skipping tests as would be expected.
>> > >
>> > > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with 
>> > > diagnostics:
>> > > Java version 8 is required for this package; found version: 11.0.1
>> > > Execution halted
>> > >
>> > > * checking PDF version of manual ... OK
>> > > * DONE
>> > > Status: 1 WARNING, 1 NOTE
>> > >
>> > > Current CRAN status: ERROR: 1, OK: 1
>> > > See: 
>> > >
>> > > Version: 2.3.0
>> > > Check: tests, Result: ERROR
>> > > Running ¡¥run-all.R¡Š [8s/35s]
>> > > Running the tests in ¡¥tests/run-all.R¡Š failed.
>> > > Last 13 lines of output:
>> > > 4: 
>> > > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
>> > > "fit", formula,
>> > > data@sdf, tolower(family$family), family$link, tol, as.integer(maxIter), 
>> > > weightCol,
>> > > regParam, as.double(var.power), as.double(link.power), 
>> > > stringIndexerOrderType,
>> > > offsetCol)
>> > > 5: invokeJava(isStatic = TRUE, className, methodName, ...)
>> > > 6: handleErrors(returnStatus, conn)
>> > > 7: stop(readString(conn))
>> > >
>> > >  testthat results 
>> > > ùù
>> > > OK: 0 SKIPPED: 0 FAILED: 2
>> > > 1. Error: create DataFrame from list or data.frame (@test_basic.R#26)
>> > > 2. Error: spark.glm and predict (@test_basic.R#58)
>> > >
>> > >
>> > >
>> > > -- Forwarded message -
>> > > Date: Mon, Nov 5, 2018, 10:12
>> > > Subject: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>> > >
>> > > Dear maintainer,
>> > >
>> > > package SparkR_2.4.0.tar.gz does not pass the incoming checks 
>> > > automatically, please see 

Re: Removing non-deprecated R methods that were deprecated in Python, Scala?

[Sean Owen]

Sounds good, remove in 3.1? I can update accordingly.

On Tue, Nov 6, 2018, 10:46 AM Reynold Xin  Maybe deprecate and remove in next version? It is bad to just remove a
> method without deprecation notice.
>
> On Tue, Nov 6, 2018 at 5:44 AM Sean Owen  wrote:
>
>> See https://github.com/apache/spark/pull/22921#discussion_r230568058
>>
>> Methods like toDegrees, toRadians, approxCountDistinct were 'renamed'
>> in Spark 2.1: deprecated, and replaced with an identical method with
>> different name. However, these weren't actually deprecated in SparkR.
>>
>> Is it an oversight that we should just correct anyway by removing, to
>> stay synced?
>> Or deprecate and retain these in Spark 3.0.0?
>>
>> Sean
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Java 11 support

[Felix Cheung]

Speaking of, get we work to support Java 11?
That will fix all the problems below.




From: Felix Cheung 
Sent: Tuesday, November 6, 2018 8:57 AM
To: Wenchen Fan
Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

We have not been able to publish to CRAN for quite some time (since 2.3.0 was 
archived - the cause is Java 11)

I think it’s ok to announce the release of 2.4.0



From: Wenchen Fan 
Sent: Tuesday, November 6, 2018 8:51 AM
To: Felix Cheung
Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

Do you mean we should have a 2.4.0 release without CRAN and then do a 2.4.1 
immediately?

On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung 
mailto:felixcheun...@hotmail.com>> wrote:
Shivaram and I were discussing.
Actually we worked with them before. Another possible approach is to remove the 
vignettes eval and all test from the source package... in the next release.



From: Matei Zaharia mailto:matei.zaha...@gmail.com>>
Sent: Tuesday, November 6, 2018 12:07 AM
To: Felix Cheung
Cc: Sean Owen; dev; Shivaram Venkataraman
Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps we 
aren’t disabling it correctly, or perhaps they can ignore this specific 
failure. +Shivaram who might have some ideas.

Matei

> On Nov 5, 2018, at 9:09 PM, Felix Cheung 
> mailto:felixcheun...@hotmail.com>> wrote:
>
> I don¡Št know what the cause is yet.
>
> The test should be skipped because of this check
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
>
> And this
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
>
> But it ran:
> callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> "fit", formula,
>
> The earlier release was achived because of Java 11+ too so this unfortunately 
> isn¡Št new.
>
>
> From: Sean Owen mailto:sro...@gmail.com>>
> Sent: Monday, November 5, 2018 7:22 PM
> To: Felix Cheung
> Cc: dev
> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>
> What can we do to get the release through? is there any way to
> circumvent these tests or otherwise hack it? or does it need a
> maintenance release?
> On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung 
> mailto:felixcheun...@hotmail.com>> wrote:
> >
> > FYI. SparkR submission failed. It seems to detect Java 11 correctly with 
> > vignettes but not skipping tests as would be expected.
> >
> > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with diagnostics:
> > Java version 8 is required for this package; found version: 11.0.1
> > Execution halted
> >
> > * checking PDF version of manual ... OK
> > * DONE
> > Status: 1 WARNING, 1 NOTE
> >
> > Current CRAN status: ERROR: 1, OK: 1
> > See: 
> >
> > Version: 2.3.0
> > Check: tests, Result: ERROR
> > Running ¡¥run-all.R¡Š [8s/35s]
> > Running the tests in ¡¥tests/run-all.R¡Š failed.
> > Last 13 lines of output:
> > 4: callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> > "fit", formula,
> > data@sdf, tolower(family$family), family$link, tol, as.integer(maxIter), 
> > weightCol,
> > regParam, as.double(var.power), as.double(link.power), 
> > stringIndexerOrderType,
> > offsetCol)
> > 5: invokeJava(isStatic = TRUE, className, methodName, ...)
> > 6: handleErrors(returnStatus, conn)
> > 7: stop(readString(conn))
> >
> >  testthat results 
> > ùù
> > OK: 0 SKIPPED: 0 FAILED: 2
> > 1. Error: create DataFrame from list or data.frame (@test_basic.R#26)
> > 2. Error: spark.glm and predict (@test_basic.R#58)
> >
> >
> >
> > -- Forwarded message -
> > Date: Mon, Nov 5, 2018, 10:12
> > Subject: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > Dear maintainer,
> >
> > package SparkR_2.4.0.tar.gz does not pass the incoming checks 
> > automatically, please see the following pre-tests:
> > Windows: 
> > 
> > Status: 1 NOTE
> > Debian: 
> > 
> > Status: 1 WARNING, 1 NOTE
> >
> > Last released version's CRAN status: ERROR: 1, OK: 1
> > See: 
> >
> > CRAN Web: 
> >
> > Please fix all problems and resubmit a fixed version via the webform.
> > If you are not sure how to fix the problems shown, please ask for help on 
> > the R-package-devel 

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

[Felix Cheung]

We have not been able to publish to CRAN for quite some time (since 2.3.0 was 
archived - the cause is Java 11)

I think it’s ok to announce the release of 2.4.0



From: Wenchen Fan 
Sent: Tuesday, November 6, 2018 8:51 AM
To: Felix Cheung
Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman
Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

Do you mean we should have a 2.4.0 release without CRAN and then do a 2.4.1 
immediately?

On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung 
mailto:felixcheun...@hotmail.com>> wrote:
Shivaram and I were discussing.
Actually we worked with them before. Another possible approach is to remove the 
vignettes eval and all test from the source package... in the next release.



From: Matei Zaharia mailto:matei.zaha...@gmail.com>>
Sent: Tuesday, November 6, 2018 12:07 AM
To: Felix Cheung
Cc: Sean Owen; dev; Shivaram Venkataraman
Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps we 
aren’t disabling it correctly, or perhaps they can ignore this specific 
failure. +Shivaram who might have some ideas.

Matei

> On Nov 5, 2018, at 9:09 PM, Felix Cheung 
> mailto:felixcheun...@hotmail.com>> wrote:
>
> I don¡Št know what the cause is yet.
>
> The test should be skipped because of this check
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
>
> And this
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
>
> But it ran:
> callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> "fit", formula,
>
> The earlier release was achived because of Java 11+ too so this unfortunately 
> isn¡Št new.
>
>
> From: Sean Owen mailto:sro...@gmail.com>>
> Sent: Monday, November 5, 2018 7:22 PM
> To: Felix Cheung
> Cc: dev
> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>
> What can we do to get the release through? is there any way to
> circumvent these tests or otherwise hack it? or does it need a
> maintenance release?
> On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung 
> mailto:felixcheun...@hotmail.com>> wrote:
> >
> > FYI. SparkR submission failed. It seems to detect Java 11 correctly with 
> > vignettes but not skipping tests as would be expected.
> >
> > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with diagnostics:
> > Java version 8 is required for this package; found version: 11.0.1
> > Execution halted
> >
> > * checking PDF version of manual ... OK
> > * DONE
> > Status: 1 WARNING, 1 NOTE
> >
> > Current CRAN status: ERROR: 1, OK: 1
> > See: 
> >
> > Version: 2.3.0
> > Check: tests, Result: ERROR
> > Running ¡¥run-all.R¡Š [8s/35s]
> > Running the tests in ¡¥tests/run-all.R¡Š failed.
> > Last 13 lines of output:
> > 4: callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> > "fit", formula,
> > data@sdf, tolower(family$family), family$link, tol, as.integer(maxIter), 
> > weightCol,
> > regParam, as.double(var.power), as.double(link.power), 
> > stringIndexerOrderType,
> > offsetCol)
> > 5: invokeJava(isStatic = TRUE, className, methodName, ...)
> > 6: handleErrors(returnStatus, conn)
> > 7: stop(readString(conn))
> >
> >  testthat results 
> > ùù
> > OK: 0 SKIPPED: 0 FAILED: 2
> > 1. Error: create DataFrame from list or data.frame (@test_basic.R#26)
> > 2. Error: spark.glm and predict (@test_basic.R#58)
> >
> >
> >
> > -- Forwarded message -
> > Date: Mon, Nov 5, 2018, 10:12
> > Subject: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > Dear maintainer,
> >
> > package SparkR_2.4.0.tar.gz does not pass the incoming checks 
> > automatically, please see the following pre-tests:
> > Windows: 
> > 
> > Status: 1 NOTE
> > Debian: 
> > 
> > Status: 1 WARNING, 1 NOTE
> >
> > Last released version's CRAN status: ERROR: 1, OK: 1
> > See: 
> >
> > CRAN Web: 
> >
> > Please fix all problems and resubmit a fixed version via the webform.
> > If you are not sure how to fix the problems shown, please ask for help on 
> > the R-package-devel mailing list:
> > 
> > If you are fairly certain the rejection is a false positive, please 
> > reply-all to this message and explain.
> >
> > More details are given in the directory:
> > 
> > 

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

[Wenchen Fan]

Do you mean we should have a 2.4.0 release without CRAN and then do a 2.4.1
immediately?

On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung 
wrote:

> Shivaram and I were discussing.
> Actually we worked with them before. Another possible approach is to
> remove the vignettes eval and all test from the source package... in the
> next release.
>
>
> --
> *From:* Matei Zaharia 
> *Sent:* Tuesday, November 6, 2018 12:07 AM
> *To:* Felix Cheung
> *Cc:* Sean Owen; dev; Shivaram Venkataraman
> *Subject:* Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>
> Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps
> we aren’t disabling it correctly, or perhaps they can ignore this specific
> failure. +Shivaram who might have some ideas.
>
> Matei
>
> > On Nov 5, 2018, at 9:09 PM, Felix Cheung 
> wrote:
> >
> > I don¡Št know what the cause is yet.
> >
> > The test should be skipped because of this check
> >
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
> >
> > And this
> >
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
> >
> > But it ran:
> > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper",
> "fit", formula,
> >
> > The earlier release was achived because of Java 11+ too so this
> unfortunately isn¡Št new.
> >
> >
> > From: Sean Owen 
> > Sent: Monday, November 5, 2018 7:22 PM
> > To: Felix Cheung
> > Cc: dev
> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > What can we do to get the release through? is there any way to
> > circumvent these tests or otherwise hack it? or does it need a
> > maintenance release?
> > On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung 
> wrote:
> > >
> > > FYI. SparkR submission failed. It seems to detect Java 11 correctly
> with vignettes but not skipping tests as would be expected.
> > >
> > > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with
> diagnostics:
> > > Java version 8 is required for this package; found version: 11.0.1
> > > Execution halted
> > >
> > > * checking PDF version of manual ... OK
> > > * DONE
> > > Status: 1 WARNING, 1 NOTE
> > >
> > > Current CRAN status: ERROR: 1, OK: 1
> > > See: 
>
> > >
> > > Version: 2.3.0
> > > Check: tests, Result: ERROR
> > > Running ¡¥run-all.R¡Š [8s/35s]
> > > Running the tests in ¡¥tests/run-all.R¡Š failed.
> > > Last 13 lines of output:
> > > 4:
> callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper",
> "fit", formula,
> > > data@sdf, tolower(family$family), family$link, tol,
> as.integer(maxIter), weightCol,
> > > regParam, as.double(var.power), as.double(link.power),
> stringIndexerOrderType,
> > > offsetCol)
> > > 5: invokeJava(isStatic = TRUE, className, methodName, ...)
> > > 6: handleErrors(returnStatus, conn)
> > > 7: stop(readString(conn))
> > >
> > >  testthat results
> ùù
>
> > > OK: 0 SKIPPED: 0 FAILED: 2
> > > 1. Error: create DataFrame from list or data.frame (@test_basic.R#26)
> > > 2. Error: spark.glm and predict (@test_basic.R#58)
> > >
> > >
> > >
> > > -- Forwarded message -
> > > Date: Mon, Nov 5, 2018, 10:12
> > > Subject: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> > >
> > > Dear maintainer,
> > >
> > > package SparkR_2.4.0.tar.gz does not pass the incoming checks
> automatically, please see the following pre-tests:
> > > Windows: <
> https://win-builder.r-project.org/incoming_pretest/SparkR_2.4.0_20181105_165757/Windows/00check.log>
>
> > > Status: 1 NOTE
> > > Debian: <
> https://win-builder.r-project.org/incoming_pretest/SparkR_2.4.0_20181105_165757/Debian/00check.log>
>
> > > Status: 1 WARNING, 1 NOTE
> > >
> > > Last released version's CRAN status: ERROR: 1, OK: 1
> > > See: 
>
> > >
> > > CRAN Web: 
> > >
> > > Please fix all problems and resubmit a fixed version via the webform.
> > > If you are not sure how to fix the problems shown, please ask for help
> on the R-package-devel mailing list:
> > > 
> > > If you are fairly certain the rejection is a false positive, please
> reply-all to this message and explain.
> > >
> > > More details are given in the directory:
> > > <
> https://win-builder.r-project.org/incoming_pretest/SparkR_2.4.0_20181105_165757/>
>
> > > The files will be removed after roughly 7 days.
> > >
> > > No strong reverse dependencies to be checked.
> > >
> > > Best regards,
> > > CRAN teams' auto-check service
> > > Flavor: r-devel-linux-x86_64-debian-gcc, r-devel-windows-ix86+x86_64
> > > Check: CRAN incoming feasibility, Result: NOTE
> > > Maintainer: 'Shivaram Venkataraman '
> > >
> > > New submission
> 

Re: Removing non-deprecated R methods that were deprecated in Python, Scala?

[Reynold Xin]

Maybe deprecate and remove in next version? It is bad to just remove a
method without deprecation notice.

On Tue, Nov 6, 2018 at 5:44 AM Sean Owen  wrote:

> See https://github.com/apache/spark/pull/22921#discussion_r230568058
>
> Methods like toDegrees, toRadians, approxCountDistinct were 'renamed'
> in Spark 2.1: deprecated, and replaced with an identical method with
> different name. However, these weren't actually deprecated in SparkR.
>
> Is it an oversight that we should just correct anyway by removing, to
> stay synced?
> Or deprecate and retain these in Spark 3.0.0?
>
> Sean
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

[Felix Cheung]

Shivaram and I were discussing.
Actually we worked with them before. Another possible approach is to remove the 
vignettes eval and all test from the source package... in the next release.



From: Matei Zaharia 
Sent: Tuesday, November 6, 2018 12:07 AM
To: Felix Cheung
Cc: Sean Owen; dev; Shivaram Venkataraman
Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps we 
aren’t disabling it correctly, or perhaps they can ignore this specific 
failure. +Shivaram who might have some ideas.

Matei

> On Nov 5, 2018, at 9:09 PM, Felix Cheung  wrote:
>
> I don¡Št know what the cause is yet.
>
> The test should be skipped because of this check
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
>
> And this
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
>
> But it ran:
> callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> "fit", formula,
>
> The earlier release was achived because of Java 11+ too so this unfortunately 
> isn¡Št new.
>
>
> From: Sean Owen 
> Sent: Monday, November 5, 2018 7:22 PM
> To: Felix Cheung
> Cc: dev
> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>
> What can we do to get the release through? is there any way to
> circumvent these tests or otherwise hack it? or does it need a
> maintenance release?
> On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung  wrote:
> >
> > FYI. SparkR submission failed. It seems to detect Java 11 correctly with 
> > vignettes but not skipping tests as would be expected.
> >
> > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with diagnostics:
> > Java version 8 is required for this package; found version: 11.0.1
> > Execution halted
> >
> > * checking PDF version of manual ... OK
> > * DONE
> > Status: 1 WARNING, 1 NOTE
> >
> > Current CRAN status: ERROR: 1, OK: 1
> > See: 
> >
> > Version: 2.3.0
> > Check: tests, Result: ERROR
> > Running ¡¥run-all.R¡Š [8s/35s]
> > Running the tests in ¡¥tests/run-all.R¡Š failed.
> > Last 13 lines of output:
> > 4: callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> > "fit", formula,
> > data@sdf, tolower(family$family), family$link, tol, as.integer(maxIter), 
> > weightCol,
> > regParam, as.double(var.power), as.double(link.power), 
> > stringIndexerOrderType,
> > offsetCol)
> > 5: invokeJava(isStatic = TRUE, className, methodName, ...)
> > 6: handleErrors(returnStatus, conn)
> > 7: stop(readString(conn))
> >
> >  testthat results 
> > ùù
> > OK: 0 SKIPPED: 0 FAILED: 2
> > 1. Error: create DataFrame from list or data.frame (@test_basic.R#26)
> > 2. Error: spark.glm and predict (@test_basic.R#58)
> >
> >
> >
> > -- Forwarded message -
> > Date: Mon, Nov 5, 2018, 10:12
> > Subject: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > Dear maintainer,
> >
> > package SparkR_2.4.0.tar.gz does not pass the incoming checks 
> > automatically, please see the following pre-tests:
> > Windows: 
> > 
> > Status: 1 NOTE
> > Debian: 
> > 
> > Status: 1 WARNING, 1 NOTE
> >
> > Last released version's CRAN status: ERROR: 1, OK: 1
> > See: 
> >
> > CRAN Web: 
> >
> > Please fix all problems and resubmit a fixed version via the webform.
> > If you are not sure how to fix the problems shown, please ask for help on 
> > the R-package-devel mailing list:
> > 
> > If you are fairly certain the rejection is a false positive, please 
> > reply-all to this message and explain.
> >
> > More details are given in the directory:
> > 
> > The files will be removed after roughly 7 days.
> >
> > No strong reverse dependencies to be checked.
> >
> > Best regards,
> > CRAN teams' auto-check service
> > Flavor: r-devel-linux-x86_64-debian-gcc, r-devel-windows-ix86+x86_64
> > Check: CRAN incoming feasibility, Result: NOTE
> > Maintainer: 'Shivaram Venkataraman '
> >
> > New submission
> >
> > Package was archived on CRAN
> >
> > Possibly mis-spelled words in DESCRIPTION:
> > Frontend (4:10, 5:28)
> >
> > CRAN repository db overrides:
> > X-CRAN-Comment: Archived on 2018-05-01 as check problems were not
> > corrected despite reminders.
> >
> > Flavor: r-devel-linux-x86_64-debian-gcc
> > Check: re-building of vignette outputs, Result: WARNING
> > 

Removing non-deprecated R methods that were deprecated in Python, Scala?

[Sean Owen]

See https://github.com/apache/spark/pull/22921#discussion_r230568058

Methods like toDegrees, toRadians, approxCountDistinct were 'renamed'
in Spark 2.1: deprecated, and replaced with an identical method with
different name. However, these weren't actually deprecated in SparkR.

Is it an oversight that we should just correct anyway by removing, to
stay synced?
Or deprecate and retain these in Spark 3.0.0?

Sean

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

[Matei Zaharia]

Maybe it’s wroth contacting the CRAN maintainers to ask for help? Perhaps we 
aren’t disabling it correctly, or perhaps they can ignore this specific 
failure. +Shivaram who might have some ideas.

Matei

> On Nov 5, 2018, at 9:09 PM, Felix Cheung  wrote:
> 
> I don¡Št know what the cause is yet.
> 
> The test should be skipped because of this check
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21
> 
> And this
> https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57
> 
> But it ran:
> callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> "fit", formula,
> 
> The earlier release was achived because of Java 11+ too so this unfortunately 
> isn¡Št new.
> 
> 
> From: Sean Owen 
> Sent: Monday, November 5, 2018 7:22 PM
> To: Felix Cheung
> Cc: dev
> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
>  
> What can we do to get the release through? is there any way to
> circumvent these tests or otherwise hack it? or does it need a
> maintenance release?
> On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung  wrote:
> >
> > FYI. SparkR submission failed. It seems to detect Java 11 correctly with 
> > vignettes but not skipping tests as would be expected.
> >
> > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with diagnostics:
> > Java version 8 is required for this package; found version: 11.0.1
> > Execution halted
> >
> > * checking PDF version of manual ... OK
> > * DONE
> > Status: 1 WARNING, 1 NOTE
> >
> > Current CRAN status: ERROR: 1, OK: 1
> > See: 
> >
> > Version: 2.3.0
> > Check: tests, Result: ERROR
> > Running ¡¥run-all.R¡Š [8s/35s]
> > Running the tests in ¡¥tests/run-all.R¡Š failed.
> > Last 13 lines of output:
> > 4: callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", 
> > "fit", formula,
> > data@sdf, tolower(family$family), family$link, tol, as.integer(maxIter), 
> > weightCol,
> > regParam, as.double(var.power), as.double(link.power), 
> > stringIndexerOrderType,
> > offsetCol)
> > 5: invokeJava(isStatic = TRUE, className, methodName, ...)
> > 6: handleErrors(returnStatus, conn)
> > 7: stop(readString(conn))
> >
> >  testthat results 
> > ùù
> > OK: 0 SKIPPED: 0 FAILED: 2
> > 1. Error: create DataFrame from list or data.frame (@test_basic.R#26)
> > 2. Error: spark.glm and predict (@test_basic.R#58)
> >
> >
> >
> > -- Forwarded message -
> > Date: Mon, Nov 5, 2018, 10:12
> > Subject: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0
> >
> > Dear maintainer,
> >
> > package SparkR_2.4.0.tar.gz does not pass the incoming checks 
> > automatically, please see the following pre-tests:
> > Windows: 
> > 
> > Status: 1 NOTE
> > Debian: 
> > 
> > Status: 1 WARNING, 1 NOTE
> >
> > Last released version's CRAN status: ERROR: 1, OK: 1
> > See: 
> >
> > CRAN Web: 
> >
> > Please fix all problems and resubmit a fixed version via the webform.
> > If you are not sure how to fix the problems shown, please ask for help on 
> > the R-package-devel mailing list:
> > 
> > If you are fairly certain the rejection is a false positive, please 
> > reply-all to this message and explain.
> >
> > More details are given in the directory:
> > 
> > The files will be removed after roughly 7 days.
> >
> > No strong reverse dependencies to be checked.
> >
> > Best regards,
> > CRAN teams' auto-check service
> > Flavor: r-devel-linux-x86_64-debian-gcc, r-devel-windows-ix86+x86_64
> > Check: CRAN incoming feasibility, Result: NOTE
> > Maintainer: 'Shivaram Venkataraman '
> >
> > New submission
> >
> > Package was archived on CRAN
> >
> > Possibly mis-spelled words in DESCRIPTION:
> > Frontend (4:10, 5:28)
> >
> > CRAN repository db overrides:
> > X-CRAN-Comment: Archived on 2018-05-01 as check problems were not
> > corrected despite reminders.
> >
> > Flavor: r-devel-linux-x86_64-debian-gcc
> > Check: re-building of vignette outputs, Result: WARNING
> > Error in re-building vignettes:
> > ...
> >
> > Attaching package: 'SparkR'
> >
> > The following objects are masked from 'package:stats':
> >
> > cov, filter, lag, na.omit, predict, sd, var, window
> >
> > The following objects are masked from 'package:base':
> >
> > as.data.frame, colnames, colnames<-, drop, endsWith,
> > intersect, rank, rbind, sample, startsWith, subset, summary,
> > transform,