I agree that's a little odd, could we not add the bacspace terminal character? Regardless even if not, I don't think that should be a blocker for 2.12 support especially since it doesn't degrade the 2.11 experience.
On Thu, Jun 7, 2018, 5:53 PM DB Tsai <d_t...@apple.com> wrote: > If we decide to initialize Spark in `initializeSynchronous()` in Scala > 2.11.12, it will look like the following which is odd. > > Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java > 1.8.0_161) > Type in expressions to have them evaluated. > Type :help for more information. > > scala> Spark context Web UI available at http://192.168.1.169:4040 > Spark context available as 'sc' (master = local[*], app id = > local-1528180279528). > Spark session available as 'spark’. > scala> > > DB Tsai | Siri Open Source Technologies [not a contribution] | > Apple, Inc > > On Jun 7, 2018, at 5:49 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > > Tests can just be changed to accept either output too :p > > On Thu, Jun 7, 2018, 5:19 PM Dean Wampler <deanwamp...@gmail.com> wrote: > >> Do the tests expect a particular console output order? That would annoy >> them. ;) You could sort the expected and output lines, then diff... >> >> >> *Dean Wampler, Ph.D.* >> >> *VP, Fast Data Engineering at Lightbend* >> Author: Programming Scala, 2nd Edition >> <http://shop.oreilly.com/product/0636920033073.do>, Fast Data >> Architectures for Streaming Applications >> <http://www.oreilly.com/data/free/fast-data-architectures-for-streaming-applications.csp>, >> and other content from O'Reilly >> @deanwampler <http://twitter.com/deanwampler> >> http://polyglotprogramming.com >> https://github.com/deanwampler >> >> On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> If the difference is the order of the welcome message I think that >>> should be fine. >>> >>> On Thu, Jun 7, 2018, 4:43 PM Dean Wampler <deanwamp...@gmail.com> wrote: >>> >>>> I'll point the Scala team to this issue, but it's unlikely to get fixed >>>> any time soon. >>>> >>>> dean >>>> >>>> >>>> *Dean Wampler, Ph.D.* >>>> >>>> *VP, Fast Data Engineering at Lightbend* >>>> Author: Programming Scala, 2nd Edition >>>> <http://shop.oreilly.com/product/0636920033073.do>, Fast Data >>>> Architectures for Streaming Applications >>>> <http://www.oreilly.com/data/free/fast-data-architectures-for-streaming-applications.csp>, >>>> and other content from O'Reilly >>>> @deanwampler <http://twitter.com/deanwampler> >>>> http://polyglotprogramming.com >>>> https://github.com/deanwampler >>>> >>>> On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai <d_t...@apple.com> wrote: >>>> >>>>> Thanks Felix for bringing this up. >>>>> >>>>> Currently, in Scala 2.11.8, we initialize the Spark by overriding >>>>> loadFIles() before REPL sees any file since there is no good hook in Scala >>>>> to load our initialization code. >>>>> >>>>> In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles() >>>>> method was removed. >>>>> >>>>> Alternatively, one way we can do in the newer version of Scala is by >>>>> overriding initializeSynchronous() suggested by Som Snytt; I have a >>>>> working >>>>> PR with this approach, >>>>> https://github.com/apache/spark/pull/21495 , and this approach should >>>>> work for older version of Scala too. >>>>> >>>>> However, in the newer version of Scala, the first thing that the REPL >>>>> calls is printWelcome, so in the newer version of Scala, welcome message >>>>> will be shown and then the URL of the SparkUI in this approach. This will >>>>> cause UI inconsistencies between different versions of Scala. >>>>> >>>>> We can also initialize the Spark in the printWelcome which I feel more >>>>> hacky. It will only work for newer version of Scala since in order version >>>>> of Scala, printWelcome is called in the end of the initialization process. >>>>> If we decide to go this route, basically users can not use Scala older >>>>> than >>>>> 2.11.9. >>>>> >>>>> I think this is also a blocker for us to move to newer version of >>>>> Scala 2.12.x since the newer version of Scala 2.12.x has the same issue. >>>>> >>>>> In my opinion, Scala should fix the root cause and provide a stable >>>>> hook for 3rd party developers to initialize their custom code. >>>>> >>>>> DB Tsai | Siri Open Source Technologies [not a contribution] | >>>>> Apple, Inc >>>>> >>>>> > On Jun 7, 2018, at 6:43 AM, Felix Cheung <felixcheun...@hotmail.com> >>>>> wrote: >>>>> > >>>>> > +1 >>>>> > >>>>> > Spoke to Dean as well and mentioned the problem with 2.11.12 >>>>> https://github.com/scala/bug/issues/10913 >>>>> > >>>>> > _____________________________ >>>>> > From: Sean Owen <sro...@gmail.com> >>>>> > Sent: Wednesday, June 6, 2018 12:23 PM >>>>> > Subject: Re: Scala 2.12 support >>>>> > To: Holden Karau <hol...@pigscanfly.ca> >>>>> > Cc: Dean Wampler <deanwamp...@gmail.com>, Reynold Xin < >>>>> r...@databricks.com>, dev <dev@spark.apache.org> >>>>> > >>>>> > >>>>> > If it means no change to 2.11 support, seems OK to me for Spark >>>>> 2.4.0. The 2.12 support is separate and has never been mutually compatible >>>>> with 2.11 builds anyway. (I also hope, suspect that the changes are >>>>> minimal; tests are already almost entirely passing with no change to the >>>>> closure cleaner when built for 2.12) >>>>> > >>>>> > On Wed, Jun 6, 2018 at 1:33 PM Holden Karau <hol...@pigscanfly.ca> >>>>> wrote: >>>>> > Just chatted with Dean @ the summit and it sounds like from Adriaan >>>>> there is a fix in 2.13 for the API change issue that could be back ported >>>>> to 2.12 so how about we try and get this ball rolling? >>>>> > >>>>> > It sounds like it would also need a closure cleaner change, which >>>>> could be backwards compatible but since it’s such a core component and we >>>>> might want to be cautious with it, we could when building for 2.11 use the >>>>> old cleaner code and for 2.12 use the new code so we don’t break anyone. >>>>> > >>>>> > How do folks feel about this? >>>>> > >>>>> > >>>>> > >>>>> >>>>> >>>> >> >
