Re: Spark on Scala 2.11
On Mon, May 12, 2014 at 6:27 PM, Matei Zaharia wrote: > We can build the REPL separately for each version of Scala, or even give > that package a different name in Scala 2.11. > OK. > Scala 2.11’s REPL actually added two flags, -Yrepl-class-based and > -Yrepl-outdir, that encompass the two modifications we made to the REPL > (using classes instead of objects to wrap each line, and grabbing the files > from some directory). So it might be possible to run it without > modifications using just a simple wrapper class around it. That would > definitely simplify things! > Exactly. I have been tracking those changes in 2.11 as well. We would need a simple wrapper around ILoop, set the Yreplclassbased flag, an HttpServer to export the repl-outdir, bind sc into repl namespace and maybe little more initialization (customizing repl init code in 2.11 is not as trivial as the <=2.10 versions, but still doable.) Still, even with all this, it should be much simpler than pulling in and refactoring everything like today. I already have a prototype for this in my working tree, still needs integration testing. > BTW did the non-REPL parts run fine on 2.11? > Currently fighting to get all the dependencies in 2.11. Quick pointer where I can get sources for akka-*-X.Y-shared-protobuf? Also, what's the smallest set of dependencies to build the smallest testable subset of the project? Thanks! > Matei > > On May 12, 2014, at 2:09 PM, Anand Avati wrote: > > > Matei, > > Thanks for confirming. I was looking specifically at the REPL part and > how > > it can be significantly simplified with 2.11 Scala, without having to > > inherit a full copy of a refactored repl inside Spark. I am happy to > > investigate/contribute a simpler 2.11 based REPL if this is were seen as > a > > priority (1.1 does not seem "too far" away.) However a 2.10 compatible > > cross build would still require a separate (existing) REPL code for the > > 2.10 build, no? > > > > Thanks. > > > > On Sun, May 11, 2014 at 2:08 PM, Matei Zaharia >wrote: > > > >> We do want to support it eventually, possibly as early as Spark 1.1 > (which > >> we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it > >> before, feel free to do so! Scala 2.11 is very close to 2.10 so I think > >> things will mostly work, except for possibly the REPL (which has require > >> porting over code form the Scala REPL in each version). > >> > >> Matei > >> > >> On May 8, 2014, at 6:33 PM, Anand Avati wrote: > >> > >>> Is there an ongoing effort (or intent) to support Spark on Scala 2.11? > >>> Approximate timeline? > >>> > >>> Thank > >> > >
Re: Spark on Scala 2.11
We can build the REPL separately for each version of Scala, or even give that package a different name in Scala 2.11. Scala 2.11’s REPL actually added two flags, -Yrepl-class-based and -Yrepl-outdir, that encompass the two modifications we made to the REPL (using classes instead of objects to wrap each line, and grabbing the files from some directory). So it might be possible to run it without modifications using just a simple wrapper class around it. That would definitely simplify things! BTW did the non-REPL parts run fine on 2.11? Matei On May 12, 2014, at 2:09 PM, Anand Avati wrote: > Matei, > Thanks for confirming. I was looking specifically at the REPL part and how > it can be significantly simplified with 2.11 Scala, without having to > inherit a full copy of a refactored repl inside Spark. I am happy to > investigate/contribute a simpler 2.11 based REPL if this is were seen as a > priority (1.1 does not seem "too far" away.) However a 2.10 compatible > cross build would still require a separate (existing) REPL code for the > 2.10 build, no? > > Thanks. > > On Sun, May 11, 2014 at 2:08 PM, Matei Zaharia wrote: > >> We do want to support it eventually, possibly as early as Spark 1.1 (which >> we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it >> before, feel free to do so! Scala 2.11 is very close to 2.10 so I think >> things will mostly work, except for possibly the REPL (which has require >> porting over code form the Scala REPL in each version). >> >> Matei >> >> On May 8, 2014, at 6:33 PM, Anand Avati wrote: >> >>> Is there an ongoing effort (or intent) to support Spark on Scala 2.11? >>> Approximate timeline? >>> >>> Thank >>
Re: Spark on Scala 2.11
Thanks a lot! Jacek On Tue, May 13, 2014 at 1:54 AM, Matei Zaharia wrote: > Anyone can actually open a JIRA on > https://issues.apache.org/jira/browse/SPARK. I’ve created one for this now: > https://issues.apache.org/jira/browse/SPARK-1812. > > Matei > > On May 12, 2014, at 3:54 PM, Jacek Laskowski wrote: > >> On Sun, May 11, 2014 at 11:08 PM, Matei Zaharia >> wrote: >>> We do want to support it eventually, possibly as early as Spark 1.1 (which >>> we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it >>> before, feel free to do so! Scala 2.11 is very close to 2.10 so I think >>> things will mostly work, except for possibly the REPL (which has require >>> porting over code form the Scala REPL in each version). >> >> Hi, >> >> Would that be possible to have a JIRA issue for this (so I could have >> a branch for the cross-build in sbt and give the task a try)? >> >> Jacek >> >> -- >> Jacek Laskowski | http://blog.japila.pl >> "Never discourage anyone who continually makes progress, no matter how >> slow." Plato > -- Jacek Laskowski | http://blog.japila.pl "Never discourage anyone who continually makes progress, no matter how slow." Plato
Re: Spark on Scala 2.11
Anyone can actually open a JIRA on https://issues.apache.org/jira/browse/SPARK. I’ve created one for this now: https://issues.apache.org/jira/browse/SPARK-1812. Matei On May 12, 2014, at 3:54 PM, Jacek Laskowski wrote: > On Sun, May 11, 2014 at 11:08 PM, Matei Zaharia > wrote: >> We do want to support it eventually, possibly as early as Spark 1.1 (which >> we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it >> before, feel free to do so! Scala 2.11 is very close to 2.10 so I think >> things will mostly work, except for possibly the REPL (which has require >> porting over code form the Scala REPL in each version). > > Hi, > > Would that be possible to have a JIRA issue for this (so I could have > a branch for the cross-build in sbt and give the task a try)? > > Jacek > > -- > Jacek Laskowski | http://blog.japila.pl > "Never discourage anyone who continually makes progress, no matter how > slow." Plato
Re: Spark on Scala 2.11
On Sun, May 11, 2014 at 11:08 PM, Matei Zaharia wrote: > We do want to support it eventually, possibly as early as Spark 1.1 (which > we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it > before, feel free to do so! Scala 2.11 is very close to 2.10 so I think > things will mostly work, except for possibly the REPL (which has require > porting over code form the Scala REPL in each version). Hi, Would that be possible to have a JIRA issue for this (so I could have a branch for the cross-build in sbt and give the task a try)? Jacek -- Jacek Laskowski | http://blog.japila.pl "Never discourage anyone who continually makes progress, no matter how slow." Plato
Re: Spark on Scala 2.11
Matei, Thanks for confirming. I was looking specifically at the REPL part and how it can be significantly simplified with 2.11 Scala, without having to inherit a full copy of a refactored repl inside Spark. I am happy to investigate/contribute a simpler 2.11 based REPL if this is were seen as a priority (1.1 does not seem "too far" away.) However a 2.10 compatible cross build would still require a separate (existing) REPL code for the 2.10 build, no? Thanks. On Sun, May 11, 2014 at 2:08 PM, Matei Zaharia wrote: > We do want to support it eventually, possibly as early as Spark 1.1 (which > we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it > before, feel free to do so! Scala 2.11 is very close to 2.10 so I think > things will mostly work, except for possibly the REPL (which has require > porting over code form the Scala REPL in each version). > > Matei > > On May 8, 2014, at 6:33 PM, Anand Avati wrote: > > > Is there an ongoing effort (or intent) to support Spark on Scala 2.11? > > Approximate timeline? > > > > Thank >
Re: Spark on Scala 2.11
i believe matei has said before that he would like to crossbuild for 2.10 and 2.11, given that the difference is not as big as between 2.9 and 2.10. but dont know when this would happen... On Sat, May 10, 2014 at 11:02 PM, Gary Malouf wrote: > Considering the team just bumped to 2.10 in 0.9, I would be surprised if > this is a near term priority. > > > On Thu, May 8, 2014 at 9:33 PM, Anand Avati wrote: > > > Is there an ongoing effort (or intent) to support Spark on Scala 2.11? > > Approximate timeline? > > > > Thanks > > >
Re: Spark on Scala 2.11
We do want to support it eventually, possibly as early as Spark 1.1 (which we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it before, feel free to do so! Scala 2.11 is very close to 2.10 so I think things will mostly work, except for possibly the REPL (which has require porting over code form the Scala REPL in each version). Matei On May 8, 2014, at 6:33 PM, Anand Avati wrote: > Is there an ongoing effort (or intent) to support Spark on Scala 2.11? > Approximate timeline? > > Thanks
Re: Spark on Scala 2.11
Considering the team just bumped to 2.10 in 0.9, I would be surprised if this is a near term priority. On Thu, May 8, 2014 at 9:33 PM, Anand Avati wrote: > Is there an ongoing effort (or intent) to support Spark on Scala 2.11? > Approximate timeline? > > Thanks >
Spark on Scala 2.11
Is there an ongoing effort (or intent) to support Spark on Scala 2.11? Approximate timeline? Thanks
