Re: Make Scala 2.12 as default Scala version in Spark 3.0

2018-11-16 Thread Justin Miller
I’d add if folks rely on Twitter in their stack, they might be stuck on older versions for a while (of their Twitter libs) which might require they stay on 2.11 for longer than they might otherwise like. On Friday, November 16, 2018, Marcelo Vanzin wrote: > Now that the switch to 2.12 by

Re: Make Scala 2.12 as default Scala version in Spark 3.0

2018-11-16 Thread Marcelo Vanzin
Now that the switch to 2.12 by default has been made, it might be good to have a serious discussion about dropping 2.11 altogether. Many of the main arguments have already been talked about. But I don't remember anyone mentioning how easy it would be to break the 2.11 build now. For example, the

Re: Is spark.sql.codegen.factoryMode property really for tests only?

2018-11-16 Thread Jacek Laskowski
Hi Marco, Many thanks for such a quick response. With that, I'll direct my curiosity into a different direction. Thanks! Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski Mastering Spark SQL https://bit.ly/mastering-spark-sql Spark Structured Streaming

Re: Is spark.sql.codegen.factoryMode property really for tests only?

2018-11-16 Thread Marco Gaido
Hi Jacek, I do believe it is correct. Please check the method you mentioned (CodeGeneratorWithInterpretedFallback.createObject): the value is relevant only if Utils.isTesting. Thanks, Marco Il giorno ven 16 nov 2018 alle ore 13:28 Jacek Laskowski ha scritto: > Hi, > > While reviewing the

Is spark.sql.codegen.factoryMode property really for tests only?

2018-11-16 Thread Jacek Laskowski
Hi, While reviewing the changes in 2.4 I stumbled upon spark.sql.codegen.factoryMode internal configuration property [1]. The doc says: > Note that this config works only for tests. Is that correct? I've got some doubts. I found that it's used in UnsafeProjection.create [2] (through