It is from the most recent 2.11 I don’t try it yet on 2.12, but I expect to get the same result.
On Thu, Jun 7, 2018 at 6:28 PM Wenchen Fan <cloud0...@gmail.com> wrote: > One more point: There was a time that we maintain 2 Spark REPL codebase > for Scala 2.10 and 2.11, maybe we can do the same for Scala 2.11 and 2.12? > if it's too hard to find a common way to do that between different Scala > versions. > > On Thu, Jun 7, 2018 at 6:20 PM, Marcelo Vanzin <van...@cloudera.com> > wrote: > >> But DB's shell output is on the most recent 2.11, not 2.12, right? >> >> On Thu, Jun 7, 2018 at 5:54 PM, Holden Karau <hol...@pigscanfly.ca> >> wrote: >> > I agree that's a little odd, could we not add the bacspace terminal >> > character? Regardless even if not, I don't think that should be a >> blocker >> > for 2.12 support especially since it doesn't degrade the 2.11 >> experience. >> > >> > On Thu, Jun 7, 2018, 5:53 PM DB Tsai <d_t...@apple.com> wrote: >> >> >> >> If we decide to initialize Spark in `initializeSynchronous()` in Scala >> >> 2.11.12, it will look like the following which is odd. >> >> >> >> Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java >> >> 1.8.0_161) >> >> Type in expressions to have them evaluated. >> >> Type :help for more information. >> >> >> >> scala> Spark context Web UI available at http://192.168.1.169:4040 >> >> Spark context available as 'sc' (master = local[*], app id = >> >> local-1528180279528). >> >> Spark session available as 'spark’. >> >> scala> >> >> >> >> DB Tsai | Siri Open Source Technologies [not a contribution] | >> >> Apple, Inc >> >> >> >> On Jun 7, 2018, at 5:49 PM, Holden Karau <hol...@pigscanfly.ca> wrote: >> >> >> >> Tests can just be changed to accept either output too :p >> >> >> >> On Thu, Jun 7, 2018, 5:19 PM Dean Wampler <deanwamp...@gmail.com> >> wrote: >> >>> >> >>> Do the tests expect a particular console output order? That would >> annoy >> >>> them. ;) You could sort the expected and output lines, then diff... >> >>> >> >>> Dean Wampler, Ph.D. >> >>> VP, Fast Data Engineering at Lightbend >> >>> Author: Programming Scala, 2nd Edition, Fast Data Architectures for >> >>> Streaming Applications, and other content from O'Reilly >> >>> @deanwampler >> >>> http://polyglotprogramming.com >> >>> https://github.com/deanwampler >> >>> >> >>> On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau <hol...@pigscanfly.ca> >> >>> wrote: >> >>>> >> >>>> If the difference is the order of the welcome message I think that >> >>>> should be fine. >> >>>> >> >>>> On Thu, Jun 7, 2018, 4:43 PM Dean Wampler <deanwamp...@gmail.com> >> wrote: >> >>>>> >> >>>>> I'll point the Scala team to this issue, but it's unlikely to get >> fixed >> >>>>> any time soon. >> >>>>> >> >>>>> dean >> >>>>> >> >>>>> Dean Wampler, Ph.D. >> >>>>> VP, Fast Data Engineering at Lightbend >> >>>>> Author: Programming Scala, 2nd Edition, Fast Data Architectures for >> >>>>> Streaming Applications, and other content from O'Reilly >> >>>>> @deanwampler >> >>>>> http://polyglotprogramming.com >> >>>>> https://github.com/deanwampler >> >>>>> >> >>>>> On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai <d_t...@apple.com> wrote: >> >>>>>> >> >>>>>> Thanks Felix for bringing this up. >> >>>>>> >> >>>>>> Currently, in Scala 2.11.8, we initialize the Spark by overriding >> >>>>>> loadFIles() before REPL sees any file since there is no good hook >> in Scala >> >>>>>> to load our initialization code. >> >>>>>> >> >>>>>> In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles() >> >>>>>> method was removed. >> >>>>>> >> >>>>>> Alternatively, one way we can do in the newer version of Scala is >> by >> >>>>>> overriding initializeSynchronous() suggested by Som Snytt; I have >> a working >> >>>>>> PR with this approach, >> >>>>>> https://github.com/apache/spark/pull/21495 , and this approach >> should >> >>>>>> work for older version of Scala too. >> >>>>>> >> >>>>>> However, in the newer version of Scala, the first thing that the >> REPL >> >>>>>> calls is printWelcome, so in the newer version of Scala, welcome >> message >> >>>>>> will be shown and then the URL of the SparkUI in this approach. >> This will >> >>>>>> cause UI inconsistencies between different versions of Scala. >> >>>>>> >> >>>>>> We can also initialize the Spark in the printWelcome which I feel >> more >> >>>>>> hacky. It will only work for newer version of Scala since in order >> version >> >>>>>> of Scala, printWelcome is called in the end of the initialization >> process. >> >>>>>> If we decide to go this route, basically users can not use Scala >> older than >> >>>>>> 2.11.9. >> >>>>>> >> >>>>>> I think this is also a blocker for us to move to newer version of >> >>>>>> Scala 2.12.x since the newer version of Scala 2.12.x has the same >> issue. >> >>>>>> >> >>>>>> In my opinion, Scala should fix the root cause and provide a stable >> >>>>>> hook for 3rd party developers to initialize their custom code. >> >>>>>> >> >>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] | >> >> >>>>>> Apple, Inc >> >>>>>> >> >>>>>> > On Jun 7, 2018, at 6:43 AM, Felix Cheung < >> felixcheun...@hotmail.com> >> >>>>>> > wrote: >> >>>>>> > >> >>>>>> > +1 >> >>>>>> > >> >>>>>> > Spoke to Dean as well and mentioned the problem with 2.11.12 >> >>>>>> > https://github.com/scala/bug/issues/10913 >> >>>>>> > >> >>>>>> > _____________________________ >> >>>>>> > From: Sean Owen <sro...@gmail.com> >> >>>>>> > Sent: Wednesday, June 6, 2018 12:23 PM >> >>>>>> > Subject: Re: Scala 2.12 support >> >>>>>> > To: Holden Karau <hol...@pigscanfly.ca> >> >>>>>> > Cc: Dean Wampler <deanwamp...@gmail.com>, Reynold Xin >> >>>>>> > <r...@databricks.com>, dev <dev@spark.apache.org> >> >>>>>> > >> >>>>>> > >> >>>>>> > If it means no change to 2.11 support, seems OK to me for Spark >> >>>>>> > 2.4.0. The 2.12 support is separate and has never been mutually >> compatible >> >>>>>> > with 2.11 builds anyway. (I also hope, suspect that the changes >> are minimal; >> >>>>>> > tests are already almost entirely passing with no change to the >> closure >> >>>>>> > cleaner when built for 2.12) >> >>>>>> > >> >>>>>> > On Wed, Jun 6, 2018 at 1:33 PM Holden Karau < >> hol...@pigscanfly.ca> >> >>>>>> > wrote: >> >>>>>> > Just chatted with Dean @ the summit and it sounds like from >> Adriaan >> >>>>>> > there is a fix in 2.13 for the API change issue that could be >> back ported to >> >>>>>> > 2.12 so how about we try and get this ball rolling? >> >>>>>> > >> >>>>>> > It sounds like it would also need a closure cleaner change, which >> >>>>>> > could be backwards compatible but since it’s such a core >> component and we >> >>>>>> > might want to be cautious with it, we could when building for >> 2.11 use the >> >>>>>> > old cleaner code and for 2.12 use the new code so we don’t break >> anyone. >> >>>>>> > >> >>>>>> > How do folks feel about this? >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> >> >>>>> >> >>> >> >> >> > >> >> >> >> -- >> Marcelo >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > -- - DB Sent from my iPhone
