Yeah, pretty tricky change but I’m glad that it’s possible to fix it. Matei
> On Sep 19, 2017, at 12:50 PM, Jacek Laskowski <ja...@japila.pl> wrote: > > Hi, > > Nice catch, Sean! Learnt this today. They did say you could learn a lot with > Spark! :) > > Pozdrawiam, > Jacek Laskowski > ---- > https://about.me/JacekLaskowski > Spark Structured Streaming (Apache Spark 2.2+) > https://bit.ly/spark-structured-streaming > Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > On Tue, Sep 19, 2017 at 4:23 PM, Sean Owen <so...@cloudera.com> wrote: > I figured this out. It's another effect of a new behavior in 2.12: > Eta-expansion of zero-argument method values is deprecated > Imagine: > > def f(): String = "foo" > def g(fn: () => String) = ??? > > g(f) works in 2.11 without warning. It generates a warning in 2.12, because > it wants you to explicitly make a function from the method reference: g(() => > f). It will maybe be an error in 2.13. > > But, this affects implicit resolution. Some of the implicits that power > SparkContext.sequenceFile() need to change to be vals of type () => > WritableConverter[T], not methods that return WritableConverter[T]. > > I'm working through this and other deprecated items in 2.12 and preparing > more 2.11-compatible changes that allow these to work cleanly in 2.12. > > On Fri, Sep 15, 2017 at 11:21 AM Sean Owen <so...@cloudera.com> wrote: > I'm working on updating to Scala 2.12, and, have hit a compile error in Scala > 2.12 that I'm strugging to design a fix to (that doesn't modify the API > significantly). If you "./dev/change-scala-version.sh 2.12" and compile, > you'll see errors like... > > [error] > /Users/srowen/Documents/Cloudera/spark/core/src/test/scala/org/apache/spark/FileSuite.scala:100: > could not find implicit value for parameter kcf: () => > org.apache.spark.WritableConverter[org.apache.hadoop.io.IntWritable] > [error] Error occurred in an application involving default arguments. > [error] val output = sc.sequenceFile[IntWritable, Text](outputDir) > > Clearly implicit resolution changed a little bit in 2.12 somehow. I actually > don't recall seeing this error before, so might be somehow related to 2.12.3, > but not sure. > > As you can see the implicits that have always existed and been imported and > should apply here don't seem to be found. > > If anyone is a Scala expert and could glance at this, you might help save me > a lot of puzzling. > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
