Re: Live Streamed Code Review today at 11am Pacific

[Holden Karau]

I'll be doing another one tomorrow morning at 9am pacific focused on Python
+ K8s support & improved JSON support -
https://www.youtube.com/watch?v=Z7ZEkvNwneU &
https://www.twitch.tv/events/xU90q9RGRGSOgp2LoNsf6A :)

On Fri, Mar 9, 2018 at 3:54 PM, Holden Karau  wrote:

> If anyone wants to watch the recording: https://www.
> youtube.com/watch?v=lugG_2QU6YU
>
> I'll do one next week as well - March 16th @ 11am -
> https://www.youtube.com/watch?v=pXzVtEUjrLc
>
> On Fri, Mar 9, 2018 at 9:28 AM, Holden Karau  wrote:
>
>> Hi folks,
>>
>> If your curious in learning more about how Spark is developed, I’m going
>> to expirement doing a live code review where folks can watch and see how
>> that part of our process works. I have two volunteers already for having
>> their PRs looked at live, and if you have a Spark PR your working on you’d
>> like me to livestream a review of please ping me.
>>
>> The livestream will be at https://www.youtube.com/watch?v=lugG_2QU6YU.
>>
>> Cheers,
>>
>> Holden :)
>> --
>> Twitter: https://twitter.com/holdenkarau
>>
>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
>



-- 
Twitter: https://twitter.com/holdenkarau

Re: Scala 2.12 support

[Sean Owen]

When I updated for Scala 2.12, I was able to remove almost all the
2.11-2.12 differences. There are still already two source trees for 2.11 vs
2.12. I mean that if it's necessary to accommodate differences between the
two, it's already set up for that, and there aren't a mess of differences
to patch over. Probably quite possible if necessary.

On Thu, Jun 7, 2018 at 8:50 PM DB Tsai  wrote:

> It is from the most recent 2.11
>
> I don’t try it yet on 2.12, but I expect to get the same result.
>
> On Thu, Jun 7, 2018 at 6:28 PM Wenchen Fan  wrote:
>
>> One more point: There was a time that we maintain 2 Spark REPL codebase
>> for Scala 2.10 and 2.11, maybe we can do the same for Scala 2.11 and 2.12?
>> if it's too hard to find a common way to do that between different Scala
>> versions.
>>
>

Re: Scala 2.12 support

[DB Tsai]

It is from the most recent 2.11

I don’t try it yet on 2.12, but I expect to get the same result.

On Thu, Jun 7, 2018 at 6:28 PM Wenchen Fan  wrote:

> One more point: There was a time that we maintain 2 Spark REPL codebase
> for Scala 2.10 and 2.11, maybe we can do the same for Scala 2.11 and 2.12?
> if it's too hard to find a common way to do that between different Scala
> versions.
>
> On Thu, Jun 7, 2018 at 6:20 PM, Marcelo Vanzin 
> wrote:
>
>> But DB's shell output is on the most recent 2.11, not 2.12, right?
>>
>> On Thu, Jun 7, 2018 at 5:54 PM, Holden Karau 
>> wrote:
>> > I agree that's a little odd, could we not add the bacspace terminal
>> > character? Regardless even if not, I don't think that should be a
>> blocker
>> > for 2.12 support especially since it doesn't degrade the 2.11
>> experience.
>> >
>> > On Thu, Jun 7, 2018, 5:53 PM DB Tsai  wrote:
>> >>
>> >> If we decide to initialize Spark in `initializeSynchronous()` in Scala
>> >> 2.11.12, it will look like the following which is odd.
>> >>
>> >> Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java
>> >> 1.8.0_161)
>> >> Type in expressions to have them evaluated.
>> >> Type :help for more information.
>> >>
>> >> scala> Spark context Web UI available at http://192.168.1.169:4040
>> >> Spark context available as 'sc' (master = local[*], app id =
>> >> local-1528180279528).
>> >> Spark session available as 'spark’.
>> >> scala>
>> >>
>> >> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
>> >> Apple, Inc
>> >>
>> >> On Jun 7, 2018, at 5:49 PM, Holden Karau  wrote:
>> >>
>> >> Tests can just be changed to accept either output too :p
>> >>
>> >> On Thu, Jun 7, 2018, 5:19 PM Dean Wampler 
>> wrote:
>> >>>
>> >>> Do the tests expect a particular console output order? That would
>> annoy
>> >>> them. ;) You could sort the expected and output lines, then diff...
>> >>>
>> >>> Dean Wampler, Ph.D.
>> >>> VP, Fast Data Engineering at Lightbend
>> >>> Author: Programming Scala, 2nd Edition, Fast Data Architectures for
>> >>> Streaming Applications, and other content from O'Reilly
>> >>> @deanwampler
>> >>> http://polyglotprogramming.com
>> >>> https://github.com/deanwampler
>> >>>
>> >>> On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau 
>> >>> wrote:
>> 
>>  If the difference is the order of the welcome message I think that
>>  should be fine.
>> 
>>  On Thu, Jun 7, 2018, 4:43 PM Dean Wampler 
>> wrote:
>> >
>> > I'll point the Scala team to this issue, but it's unlikely to get
>> fixed
>> > any time soon.
>> >
>> > dean
>> >
>> > Dean Wampler, Ph.D.
>> > VP, Fast Data Engineering at Lightbend
>> > Author: Programming Scala, 2nd Edition, Fast Data Architectures for
>> > Streaming Applications, and other content from O'Reilly
>> > @deanwampler
>> > http://polyglotprogramming.com
>> > https://github.com/deanwampler
>> >
>> > On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai  wrote:
>> >>
>> >> Thanks Felix for bringing this up.
>> >>
>> >> Currently, in Scala 2.11.8, we initialize the Spark by overriding
>> >> loadFIles() before REPL sees any file since there is no good hook
>> in Scala
>> >> to load our initialization code.
>> >>
>> >> In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles()
>> >> method was removed.
>> >>
>> >> Alternatively, one way we can do in the newer version of Scala is
>> by
>> >> overriding initializeSynchronous() suggested by Som Snytt; I have
>> a working
>> >> PR with this approach,
>> >> https://github.com/apache/spark/pull/21495 , and this approach
>> should
>> >> work for older version of Scala too.
>> >>
>> >> However, in the newer version of Scala, the first thing that the
>> REPL
>> >> calls is printWelcome, so in the newer version of Scala, welcome
>> message
>> >> will be shown and then the URL of the SparkUI in this approach.
>> This will
>> >> cause UI inconsistencies between different versions of Scala.
>> >>
>> >> We can also initialize the Spark in the printWelcome which I feel
>> more
>> >> hacky. It will only work for newer version of Scala since in order
>> version
>> >> of Scala, printWelcome is called in the end of the initialization
>> process.
>> >> If we decide to go this route, basically users can not use Scala
>> older than
>> >> 2.11.9.
>> >>
>> >> I think this is also a blocker for us to move to newer version of
>> >> Scala 2.12.x since the newer version of Scala 2.12.x has the same
>> issue.
>> >>
>> >> In my opinion, Scala should fix the root cause and provide a stable
>> >> hook for 3rd party developers to initialize their custom code.
>> >>
>> >> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |
>> 
>> >> Apple, Inc
>> >>
>> >> > On Jun 7, 2018, at 6:43 AM, Felix Cheung <
>> felixcheun...@hotmail.com>
>> >> > 

Re: Scala 2.12 support

[Holden Karau]

I agree that's a little odd, could we not add the bacspace terminal
character? Regardless even if not, I don't think that should be a blocker
for 2.12 support especially since it doesn't degrade the 2.11 experience.

On Thu, Jun 7, 2018, 5:53 PM DB Tsai  wrote:

> If we decide to initialize Spark in `initializeSynchronous()` in Scala
> 2.11.12, it will look like the following which is odd.
>
> Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.8.0_161)
> Type in expressions to have them evaluated.
> Type :help for more information.
>
> scala> Spark context Web UI available at http://192.168.1.169:4040
> Spark context available as 'sc' (master = local[*], app id =
> local-1528180279528).
> Spark session available as 'spark’.
> scala>
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
>
> On Jun 7, 2018, at 5:49 PM, Holden Karau  wrote:
>
> Tests can just be changed to accept either output too :p
>
> On Thu, Jun 7, 2018, 5:19 PM Dean Wampler  wrote:
>
>> Do the tests expect a particular console output order? That would annoy
>> them. ;) You could sort the expected and output lines, then diff...
>>
>>
>> *Dean Wampler, Ph.D.*
>>
>> *VP, Fast Data Engineering at Lightbend*
>> Author: Programming Scala, 2nd Edition
>> , Fast Data
>> Architectures for Streaming Applications
>> ,
>> and other content from O'Reilly
>> @deanwampler 
>> http://polyglotprogramming.com
>> https://github.com/deanwampler
>>
>> On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau 
>> wrote:
>>
>>> If the difference is the order of the welcome message I think that
>>> should be fine.
>>>
>>> On Thu, Jun 7, 2018, 4:43 PM Dean Wampler  wrote:
>>>
 I'll point the Scala team to this issue, but it's unlikely to get fixed
 any time soon.

 dean


 *Dean Wampler, Ph.D.*

 *VP, Fast Data Engineering at Lightbend*
 Author: Programming Scala, 2nd Edition
 , Fast Data
 Architectures for Streaming Applications
 ,
 and other content from O'Reilly
 @deanwampler 
 http://polyglotprogramming.com
 https://github.com/deanwampler

 On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai  wrote:

> Thanks Felix for bringing this up.
>
> Currently, in Scala 2.11.8, we initialize the Spark by overriding
> loadFIles() before REPL sees any file since there is no good hook in Scala
> to load our initialization code.
>
> In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles()
> method was removed.
>
> Alternatively, one way we can do in the newer version of Scala is by
> overriding initializeSynchronous() suggested by Som Snytt; I have a 
> working
> PR with this approach,
> https://github.com/apache/spark/pull/21495 , and this approach should
> work for older version of Scala too.
>
> However, in the newer version of Scala, the first thing that the REPL
> calls is printWelcome, so in the newer version of Scala, welcome message
> will be shown and then the URL of the SparkUI in this approach. This will
> cause UI inconsistencies between different versions of Scala.
>
> We can also initialize the Spark in the printWelcome which I feel more
> hacky. It will only work for newer version of Scala since in order version
> of Scala, printWelcome is called in the end of the initialization process.
> If we decide to go this route, basically users can not use Scala older 
> than
> 2.11.9.
>
> I think this is also a blocker for us to move to newer version of
> Scala 2.12.x since the newer version of Scala 2.12.x has the same issue.
>
> In my opinion, Scala should fix the root cause and provide a stable
> hook for 3rd party developers to initialize their custom code.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
>
> > On Jun 7, 2018, at 6:43 AM, Felix Cheung 
> wrote:
> >
> > +1
> >
> > Spoke to Dean as well and mentioned the problem with 2.11.12
> https://github.com/scala/bug/issues/10913
> >
> > _
> > From: Sean Owen 
> > Sent: Wednesday, June 6, 2018 12:23 PM
> > Subject: Re: Scala 2.12 support
> > To: Holden Karau 
> > Cc: Dean Wampler , Reynold Xin <
> r...@databricks.com>, dev 
> >
> >
> > If it means no change to 2.11 support, seems OK to me for Spark
> 2.4.0. The 2.12 support is separate and has never been mutually compatible
> with 2.11 builds anyway. (I also hope, suspect that the changes are
> minimal; tests are 

Re: Scala 2.12 support

[DB Tsai]

If we decide to initialize Spark in `initializeSynchronous()` in Scala 2.11.12, 
it will look like the following which is odd.

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)
Type in expressions to have them evaluated.
Type :help for more information.

scala> Spark context Web UI available at http://192.168.1.169:4040
Spark context available as 'sc' (master = local[*], app id = 
local-1528180279528).
Spark session available as 'spark’.
scala>

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Jun 7, 2018, at 5:49 PM, Holden Karau  wrote:
> 
> Tests can just be changed to accept either output too :p
> 
> On Thu, Jun 7, 2018, 5:19 PM Dean Wampler  > wrote:
> Do the tests expect a particular console output order? That would annoy them. 
> ;) You could sort the expected and output lines, then diff...
> 
> Dean Wampler, Ph.D.
> VP, Fast Data Engineering at Lightbend
> Author: Programming Scala, 2nd Edition 
> , Fast Data Architectures 
> for Streaming Applications 
> ,
>  and other content from O'Reilly
> @deanwampler 
> http://polyglotprogramming.com 
> https://github.com/deanwampler 
> 
> On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau  > wrote:
> If the difference is the order of the welcome message I think that should be 
> fine.
> 
> On Thu, Jun 7, 2018, 4:43 PM Dean Wampler  > wrote:
> I'll point the Scala team to this issue, but it's unlikely to get fixed any 
> time soon.
> 
> dean
> 
> Dean Wampler, Ph.D.
> VP, Fast Data Engineering at Lightbend
> Author: Programming Scala, 2nd Edition 
> , Fast Data Architectures 
> for Streaming Applications 
> ,
>  and other content from O'Reilly
> @deanwampler 
> http://polyglotprogramming.com 
> https://github.com/deanwampler 
> 
> On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai  > wrote:
> Thanks Felix for bringing this up.
> 
> Currently, in Scala 2.11.8, we initialize the Spark by overriding loadFIles() 
> before REPL sees any file since there is no good hook in Scala to load our 
> initialization code.
> 
> In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles() method 
> was removed.
> 
> Alternatively, one way we can do in the newer version of Scala is by 
> overriding initializeSynchronous() suggested by Som Snytt; I have a working 
> PR with this approach,
> https://github.com/apache/spark/pull/21495 
>  , and this approach should work 
> for older version of Scala too. 
> 
> However, in the newer version of Scala, the first thing that the REPL calls 
> is printWelcome, so in the newer version of Scala, welcome message will be 
> shown and then the URL of the SparkUI in this approach. This will cause UI 
> inconsistencies between different versions of Scala.
> 
> We can also initialize the Spark in the printWelcome which I feel more hacky. 
> It will only work for newer version of Scala since in order version of Scala, 
> printWelcome is called in the end of the initialization process. If we decide 
> to go this route, basically users can not use Scala older than 2.11.9.
> 
> I think this is also a blocker for us to move to newer version of Scala 
> 2.12.x since the newer version of Scala 2.12.x has the same issue.
> 
> In my opinion, Scala should fix the root cause and provide a stable hook for 
> 3rd party developers to initialize their custom code.
> 
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
> Inc
> 
> > On Jun 7, 2018, at 6:43 AM, Felix Cheung  > > wrote:
> > 
> > +1
> > 
> > Spoke to Dean as well and mentioned the problem with 2.11.12 
> > https://github.com/scala/bug/issues/10913 
> > 
> > 
> > _
> > From: Sean Owen mailto:sro...@gmail.com>>
> > Sent: Wednesday, June 6, 2018 12:23 PM
> > Subject: Re: Scala 2.12 support
> > To: Holden Karau mailto:hol...@pigscanfly.ca>>
> > Cc: Dean Wampler mailto:deanwamp...@gmail.com>>, 
> > Reynold Xin mailto:r...@databricks.com>>, dev 
> > mailto:dev@spark.apache.org>>
> > 
> > 
> > If it means no change to 2.11 support, seems OK to me for Spark 2.4.0. The 
> > 2.12 support is separate and has never been mutually compatible with 2.11 
> > builds anyway. (I also hope, suspect that the changes are minimal; tests 
> > are already almost entirely passing with no change to the closure 

Re: Scala 2.12 support

[Holden Karau]

Tests can just be changed to accept either output too :p

On Thu, Jun 7, 2018, 5:19 PM Dean Wampler  wrote:

> Do the tests expect a particular console output order? That would annoy
> them. ;) You could sort the expected and output lines, then diff...
>
>
> *Dean Wampler, Ph.D.*
>
> *VP, Fast Data Engineering at Lightbend*
> Author: Programming Scala, 2nd Edition
> , Fast Data
> Architectures for Streaming Applications
> ,
> and other content from O'Reilly
> @deanwampler 
> http://polyglotprogramming.com
> https://github.com/deanwampler
>
> On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau  wrote:
>
>> If the difference is the order of the welcome message I think that should
>> be fine.
>>
>> On Thu, Jun 7, 2018, 4:43 PM Dean Wampler  wrote:
>>
>>> I'll point the Scala team to this issue, but it's unlikely to get fixed
>>> any time soon.
>>>
>>> dean
>>>
>>>
>>> *Dean Wampler, Ph.D.*
>>>
>>> *VP, Fast Data Engineering at Lightbend*
>>> Author: Programming Scala, 2nd Edition
>>> , Fast Data
>>> Architectures for Streaming Applications
>>> ,
>>> and other content from O'Reilly
>>> @deanwampler 
>>> http://polyglotprogramming.com
>>> https://github.com/deanwampler
>>>
>>> On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai  wrote:
>>>
 Thanks Felix for bringing this up.

 Currently, in Scala 2.11.8, we initialize the Spark by overriding
 loadFIles() before REPL sees any file since there is no good hook in Scala
 to load our initialization code.

 In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles()
 method was removed.

 Alternatively, one way we can do in the newer version of Scala is by
 overriding initializeSynchronous() suggested by Som Snytt; I have a working
 PR with this approach,
 https://github.com/apache/spark/pull/21495 , and this approach should
 work for older version of Scala too.

 However, in the newer version of Scala, the first thing that the REPL
 calls is printWelcome, so in the newer version of Scala, welcome message
 will be shown and then the URL of the SparkUI in this approach. This will
 cause UI inconsistencies between different versions of Scala.

 We can also initialize the Spark in the printWelcome which I feel more
 hacky. It will only work for newer version of Scala since in order version
 of Scala, printWelcome is called in the end of the initialization process.
 If we decide to go this route, basically users can not use Scala older than
 2.11.9.

 I think this is also a blocker for us to move to newer version of Scala
 2.12.x since the newer version of Scala 2.12.x has the same issue.

 In my opinion, Scala should fix the root cause and provide a stable
 hook for 3rd party developers to initialize their custom code.

 DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
 Apple, Inc

 > On Jun 7, 2018, at 6:43 AM, Felix Cheung 
 wrote:
 >
 > +1
 >
 > Spoke to Dean as well and mentioned the problem with 2.11.12
 https://github.com/scala/bug/issues/10913
 >
 > _
 > From: Sean Owen 
 > Sent: Wednesday, June 6, 2018 12:23 PM
 > Subject: Re: Scala 2.12 support
 > To: Holden Karau 
 > Cc: Dean Wampler , Reynold Xin <
 r...@databricks.com>, dev 
 >
 >
 > If it means no change to 2.11 support, seems OK to me for Spark
 2.4.0. The 2.12 support is separate and has never been mutually compatible
 with 2.11 builds anyway. (I also hope, suspect that the changes are
 minimal; tests are already almost entirely passing with no change to the
 closure cleaner when built for 2.12)
 >
 > On Wed, Jun 6, 2018 at 1:33 PM Holden Karau 
 wrote:
 > Just chatted with Dean @ the summit and it sounds like from Adriaan
 there is a fix in 2.13 for the API change issue that could be back ported
 to 2.12 so how about we try and get this ball rolling?
 >
 > It sounds like it would also need a closure cleaner change, which
 could be backwards compatible but since it’s such a core component and we
 might want to be cautious with it, we could when building for 2.11 use the
 old cleaner code and for 2.12 use the new code so we don’t break anyone.
 >
 > How do folks feel about this?
 >
 >
 >


>>>
>

Re: Scala 2.12 support

[Dean Wampler]

Do the tests expect a particular console output order? That would annoy
them. ;) You could sort the expected and output lines, then diff...


*Dean Wampler, Ph.D.*

*VP, Fast Data Engineering at Lightbend*
Author: Programming Scala, 2nd Edition
, Fast Data Architectures
for Streaming Applications
,
and other content from O'Reilly
@deanwampler 
http://polyglotprogramming.com
https://github.com/deanwampler

On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau  wrote:

> If the difference is the order of the welcome message I think that should
> be fine.
>
> On Thu, Jun 7, 2018, 4:43 PM Dean Wampler  wrote:
>
>> I'll point the Scala team to this issue, but it's unlikely to get fixed
>> any time soon.
>>
>> dean
>>
>>
>> *Dean Wampler, Ph.D.*
>>
>> *VP, Fast Data Engineering at Lightbend*
>> Author: Programming Scala, 2nd Edition
>> , Fast Data
>> Architectures for Streaming Applications
>> ,
>> and other content from O'Reilly
>> @deanwampler 
>> http://polyglotprogramming.com
>> https://github.com/deanwampler
>>
>> On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai  wrote:
>>
>>> Thanks Felix for bringing this up.
>>>
>>> Currently, in Scala 2.11.8, we initialize the Spark by overriding
>>> loadFIles() before REPL sees any file since there is no good hook in Scala
>>> to load our initialization code.
>>>
>>> In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles()
>>> method was removed.
>>>
>>> Alternatively, one way we can do in the newer version of Scala is by
>>> overriding initializeSynchronous() suggested by Som Snytt; I have a working
>>> PR with this approach,
>>> https://github.com/apache/spark/pull/21495 , and this approach should
>>> work for older version of Scala too.
>>>
>>> However, in the newer version of Scala, the first thing that the REPL
>>> calls is printWelcome, so in the newer version of Scala, welcome message
>>> will be shown and then the URL of the SparkUI in this approach. This will
>>> cause UI inconsistencies between different versions of Scala.
>>>
>>> We can also initialize the Spark in the printWelcome which I feel more
>>> hacky. It will only work for newer version of Scala since in order version
>>> of Scala, printWelcome is called in the end of the initialization process.
>>> If we decide to go this route, basically users can not use Scala older than
>>> 2.11.9.
>>>
>>> I think this is also a blocker for us to move to newer version of Scala
>>> 2.12.x since the newer version of Scala 2.12.x has the same issue.
>>>
>>> In my opinion, Scala should fix the root cause and provide a stable hook
>>> for 3rd party developers to initialize their custom code.
>>>
>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
>>> Apple, Inc
>>>
>>> > On Jun 7, 2018, at 6:43 AM, Felix Cheung 
>>> wrote:
>>> >
>>> > +1
>>> >
>>> > Spoke to Dean as well and mentioned the problem with 2.11.12
>>> https://github.com/scala/bug/issues/10913
>>> >
>>> > _
>>> > From: Sean Owen 
>>> > Sent: Wednesday, June 6, 2018 12:23 PM
>>> > Subject: Re: Scala 2.12 support
>>> > To: Holden Karau 
>>> > Cc: Dean Wampler , Reynold Xin <
>>> r...@databricks.com>, dev 
>>> >
>>> >
>>> > If it means no change to 2.11 support, seems OK to me for Spark 2.4.0.
>>> The 2.12 support is separate and has never been mutually compatible with
>>> 2.11 builds anyway. (I also hope, suspect that the changes are minimal;
>>> tests are already almost entirely passing with no change to the closure
>>> cleaner when built for 2.12)
>>> >
>>> > On Wed, Jun 6, 2018 at 1:33 PM Holden Karau 
>>> wrote:
>>> > Just chatted with Dean @ the summit and it sounds like from Adriaan
>>> there is a fix in 2.13 for the API change issue that could be back ported
>>> to 2.12 so how about we try and get this ball rolling?
>>> >
>>> > It sounds like it would also need a closure cleaner change, which
>>> could be backwards compatible but since it’s such a core component and we
>>> might want to be cautious with it, we could when building for 2.11 use the
>>> old cleaner code and for 2.12 use the new code so we don’t break anyone.
>>> >
>>> > How do folks feel about this?
>>> >
>>> >
>>> >
>>>
>>>
>>

Re: Scala 2.12 support

[Dean Wampler]

I'll point the Scala team to this issue, but it's unlikely to get fixed any
time soon.

dean


*Dean Wampler, Ph.D.*

*VP, Fast Data Engineering at Lightbend*
Author: Programming Scala, 2nd Edition
, Fast Data Architectures
for Streaming Applications
,
and other content from O'Reilly
@deanwampler 
http://polyglotprogramming.com
https://github.com/deanwampler

On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai  wrote:

> Thanks Felix for bringing this up.
>
> Currently, in Scala 2.11.8, we initialize the Spark by overriding
> loadFIles() before REPL sees any file since there is no good hook in Scala
> to load our initialization code.
>
> In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles() method
> was removed.
>
> Alternatively, one way we can do in the newer version of Scala is by
> overriding initializeSynchronous() suggested by Som Snytt; I have a working
> PR with this approach,
> https://github.com/apache/spark/pull/21495 , and this approach should
> work for older version of Scala too.
>
> However, in the newer version of Scala, the first thing that the REPL
> calls is printWelcome, so in the newer version of Scala, welcome message
> will be shown and then the URL of the SparkUI in this approach. This will
> cause UI inconsistencies between different versions of Scala.
>
> We can also initialize the Spark in the printWelcome which I feel more
> hacky. It will only work for newer version of Scala since in order version
> of Scala, printWelcome is called in the end of the initialization process.
> If we decide to go this route, basically users can not use Scala older than
> 2.11.9.
>
> I think this is also a blocker for us to move to newer version of Scala
> 2.12.x since the newer version of Scala 2.12.x has the same issue.
>
> In my opinion, Scala should fix the root cause and provide a stable hook
> for 3rd party developers to initialize their custom code.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
>
> > On Jun 7, 2018, at 6:43 AM, Felix Cheung 
> wrote:
> >
> > +1
> >
> > Spoke to Dean as well and mentioned the problem with 2.11.12
> https://github.com/scala/bug/issues/10913
> >
> > _
> > From: Sean Owen 
> > Sent: Wednesday, June 6, 2018 12:23 PM
> > Subject: Re: Scala 2.12 support
> > To: Holden Karau 
> > Cc: Dean Wampler , Reynold Xin <
> r...@databricks.com>, dev 
> >
> >
> > If it means no change to 2.11 support, seems OK to me for Spark 2.4.0.
> The 2.12 support is separate and has never been mutually compatible with
> 2.11 builds anyway. (I also hope, suspect that the changes are minimal;
> tests are already almost entirely passing with no change to the closure
> cleaner when built for 2.12)
> >
> > On Wed, Jun 6, 2018 at 1:33 PM Holden Karau 
> wrote:
> > Just chatted with Dean @ the summit and it sounds like from Adriaan
> there is a fix in 2.13 for the API change issue that could be back ported
> to 2.12 so how about we try and get this ball rolling?
> >
> > It sounds like it would also need a closure cleaner change, which could
> be backwards compatible but since it’s such a core component and we might
> want to be cautious with it, we could when building for 2.11 use the old
> cleaner code and for 2.12 use the new code so we don’t break anyone.
> >
> > How do folks feel about this?
> >
> >
> >
>
>

Re: Scala 2.12 support

[DB Tsai]

Thanks Felix for bringing this up.

Currently, in Scala 2.11.8, we initialize the Spark by overriding loadFIles() 
before REPL sees any file since there is no good hook in Scala to load our 
initialization code.

In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles() method was 
removed.

Alternatively, one way we can do in the newer version of Scala is by overriding 
initializeSynchronous() suggested by Som Snytt; I have a working PR with this 
approach,
https://github.com/apache/spark/pull/21495 , and this approach should work for 
older version of Scala too. 

However, in the newer version of Scala, the first thing that the REPL calls is 
printWelcome, so in the newer version of Scala, welcome message will be shown 
and then the URL of the SparkUI in this approach. This will cause UI 
inconsistencies between different versions of Scala.

We can also initialize the Spark in the printWelcome which I feel more hacky. 
It will only work for newer version of Scala since in order version of Scala, 
printWelcome is called in the end of the initialization process. If we decide 
to go this route, basically users can not use Scala older than 2.11.9.

I think this is also a blocker for us to move to newer version of Scala 2.12.x 
since the newer version of Scala 2.12.x has the same issue.

In my opinion, Scala should fix the root cause and provide a stable hook for 
3rd party developers to initialize their custom code.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Jun 7, 2018, at 6:43 AM, Felix Cheung  wrote:
> 
> +1
> 
> Spoke to Dean as well and mentioned the problem with 2.11.12 
> https://github.com/scala/bug/issues/10913
> 
> _
> From: Sean Owen 
> Sent: Wednesday, June 6, 2018 12:23 PM
> Subject: Re: Scala 2.12 support
> To: Holden Karau 
> Cc: Dean Wampler , Reynold Xin , 
> dev 
> 
> 
> If it means no change to 2.11 support, seems OK to me for Spark 2.4.0. The 
> 2.12 support is separate and has never been mutually compatible with 2.11 
> builds anyway. (I also hope, suspect that the changes are minimal; tests are 
> already almost entirely passing with no change to the closure cleaner when 
> built for 2.12)
> 
> On Wed, Jun 6, 2018 at 1:33 PM Holden Karau  wrote:
> Just chatted with Dean @ the summit and it sounds like from Adriaan there is 
> a fix in 2.13 for the API change issue that could be back ported to 2.12 so 
> how about we try and get this ball rolling?
> 
> It sounds like it would also need a closure cleaner change, which could be 
> backwards compatible but since it’s such a core component and we might want 
> to be cautious with it, we could when building for 2.11 use the old cleaner 
> code and for 2.12 use the new code so we don’t break anyone.
> 
> How do folks feel about this?
> 
> 
> 


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Time for 2.2.2 release

[Marcelo Vanzin]

Took a look at our branch and most of the stuff that is not already in
2.2 are flaky test fixes, so +1.

On Wed, Jun 6, 2018 at 7:54 AM, Tom Graves  wrote:
> Hello all,
>
> I think its time for another 2.2 release.
> I took a look at Jira and I don't see anything explicitly targeted for 2.2.2
> that is not yet complete.
>
> So I'd like to propose to release 2.2.2 soon. If there are important
> fixes that should go into the release, please let those be known (by
> replying here or updating the bug in Jira), otherwise I'm volunteering
> to prepare the first RC soon-ish (by early next week since Spark Summit is
> this week).
>
> Thanks!
> Tom Graves
>



-- 
Marcelo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Scala 2.12 support

[Felix Cheung]

+1

Spoke to Dean as well and mentioned the problem with 2.11.12 
https://github.com/scala/bug/issues/10913

_
From: Sean Owen 
Sent: Wednesday, June 6, 2018 12:23 PM
Subject: Re: Scala 2.12 support
To: Holden Karau 
Cc: Dean Wampler , Reynold Xin , 
dev 


If it means no change to 2.11 support, seems OK to me for Spark 2.4.0. The 2.12 
support is separate and has never been mutually compatible with 2.11 builds 
anyway. (I also hope, suspect that the changes are minimal; tests are already 
almost entirely passing with no change to the closure cleaner when built for 
2.12)

On Wed, Jun 6, 2018 at 1:33 PM Holden Karau 
mailto:hol...@pigscanfly.ca>> wrote:
Just chatted with Dean @ the summit and it sounds like from Adriaan there is a 
fix in 2.13 for the API change issue that could be back ported to 2.12 so how 
about we try and get this ball rolling?

It sounds like it would also need a closure cleaner change, which could be 
backwards compatible but since it’s such a core component and we might want to 
be cautious with it, we could when building for 2.11 use the old cleaner code 
and for 2.12 use the new code so we don’t break anyone.

How do folks feel about this?

Re: Time for 2.2.2 release

[Felix Cheung]

+1 and thanks!


From: Tom Graves 
Sent: Wednesday, June 6, 2018 7:54:43 AM
To: Dev
Subject: Time for 2.2.2 release

Hello all,

I think its time for another 2.2 release.
I took a look at Jira and I don't see anything explicitly targeted for 2.2.2 
that is not yet complete.

So I'd like to propose to release 2.2.2 soon. If there are important
fixes that should go into the release, please let those be known (by
replying here or updating the bug in Jira), otherwise I'm volunteering
to prepare the first RC soon-ish (by early next week since Spark Summit is this 
week).

Thanks!
Tom Graves

Re: [build system] meet your build engineer @ spark ai summit SF 2018

[Hyukjin Kwon]

I regret that I couldn't make it to Spark Summit :(.

2018년 6월 6일 (수) 오전 3:25, Holden Karau 님이 작성:

> That's awesome!
>
> On Tue, Jun 5, 2018 at 12:23 PM, shane knapp  wrote:
>
>> just a reminder to come meet your build engineer!
>>
>> we'll also be having a couple of demos of current projects in the lab:
>> tuesday (today) 130pm -- pandas on ray (
>> https://rise.cs.berkeley.edu/blog/pandas-on-ray/)
>> wednesday (tomorrow) 130pm -- ray (http://ray.readthedocs.io/)
>>
>> shane
>>
>> On Wed, May 2, 2018 at 11:11 AM, shane knapp  wrote:
>>
>>> hey everyone!
>>>
>>> if you ever wanted to meet the one-man operation that keeps things
>>> going, talk about future build system plans, complain about the fact that
>>> we're still on centos 6 (yes, i know), or just say hi, i'll be manning the
>>> RISELab booth at summit all three days!
>>>
>>> :)
>>>
>>> shane
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
>