Please import: import org.apache.spark.sql.functions._
On Fri, Mar 4, 2016 at 3:35 PM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > thanks. It is like war of attrition. I always thought that you add import > before the class itself not within the class? w3hat is the reason for it > please? > > this is my code > > import org.apache.spark.SparkContext > import org.apache.spark.SparkConf > import org.apache.spark.sql.Row > import org.apache.spark.sql.hive.HiveContext > import org.apache.spark.sql.types._ > import org.apache.spark.sql.SQLContext > // > object Sequence { > def main(args: Array[String]) { > val conf = new > SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts", > "true") > val sc = new SparkContext(conf) > // Note that this should be done only after an instance of > org.apache.spark.sql.SQLContext is created. It should be written as: > val sqlContext= new org.apache.spark.sql.SQLContext(sc) > import sqlContext.implicits._ > val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) > val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16)) > // Sort option 1 using tempTable > val b = a.toDF("Name","score").registerTempTable("tmp") > HiveContext.sql("select Name,score from tmp order by score desc").show > // Sort option 2 with FP > a.toDF("Name","score").sort(desc("score")).show > } > } > > And now the last failure is in > > info] [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (104ms) > [info] Done updating. > [info] Compiling 1 Scala source to > /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes... > [info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling... > [info] Compilation completed in 15.779 s > [error] > /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:21: not > found: value desc > [error] a.toDF("Name","score").sort(desc("score")).show > [error] ^ > [error] one error found > [error] (compile:compileIncremental) Compilation failed > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 4 March 2016 at 23:25, Chandeep Singh <c...@chandeep.com> wrote: > >> This is what you need: >> >> val sc = new SparkContext(sparkConf) >> val sqlContext = new org.apache.spark.sql.SQLContext(sc) >> import sqlContext.implicits._ >> >> On Mar 4, 2016, at 11:03 PM, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >> Hi Ted, >> >> This is my code >> >> import org.apache.spark.SparkConf >> import org.apache.spark.sql.Row >> import org.apache.spark.sql.hive.HiveContext >> import org.apache.spark.sql.types._ >> import org.apache.spark.sql.SQLContext >> // >> object Sequence { >> def main(args: Array[String]) { >> val conf = new >> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts", >> "true") >> val sc = new SparkContext(conf) >> val sqlContext = new org.apache.spark.sql.SQLContext(sc) >> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) >> val a = Seq(("Mich",20), ("Christian", 18), ("James",13), >> ("Richard",16)) >> // Sort option 1 using tempTable >> val b = a.toDF("Name","score").registerTempTable("tmp") >> sql("select Name,score from tmp order by score desc").show >> // Sort option 2 with FP >> a.toDF("Name","score").sort(desc("score")).show >> } >> } >> >> And the error I am getting now is >> >> [info] downloading >> https://repo1.maven.org/maven2/org/scala-lang/jline/2.10.5/jline-2.10.5.jar >> ... >> [info] [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (103ms) >> [info] Done updating. >> [info] Compiling 1 Scala source to >> /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes... >> [info] 'compiler-interface' not yet compiled for Scala 2.10.5. >> Compiling... >> [info] Compilation completed in 12.462 s >> [error] >> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: value >> toDF is not a member of Seq[(String, Int)] >> [error] val b = a.toDF("Name","score").registerTempTable("tmp") >> [error] ^ >> [error] >> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:17: not >> found: value sql >> [error] sql("select Name,score from tmp order by score desc").show >> [error] ^ >> [error] >> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:19: value >> toDF is not a member of Seq[(String, Int)] >> [error] a.toDF("Name","score").sort(desc("score")).show >> [error] ^ >> [error] three errors found >> [error] (compile:compileIncremental) Compilation failed >> [error] Total time: 88 s, completed Mar 4, 2016 11:12:46 PM >> >> >> Dr Mich Talebzadeh >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> On 4 March 2016 at 22:52, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Can you show your code snippet ? >>> Here is an example: >>> >>> val sqlContext = new SQLContext(sc) >>> import sqlContext.implicits._ >>> >>> On Fri, Mar 4, 2016 at 1:55 PM, Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>>> Hi Ted, >>>> >>>> I am getting the following error after adding that import >>>> >>>> [error] >>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:5: not >>>> found: object sqlContext >>>> [error] import sqlContext.implicits._ >>>> [error] ^ >>>> [error] >>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value >>>> toDF is not a member of Seq[(String, Int)] >>>> >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> >>>> On 4 March 2016 at 21:39, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>>> Can you add the following into your code ? >>>>> import sqlContext.implicits._ >>>>> >>>>> On Fri, Mar 4, 2016 at 1:14 PM, Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have a simple Scala program as below >>>>>> >>>>>> import org.apache.spark.SparkContext >>>>>> import org.apache.spark.SparkContext._ >>>>>> import org.apache.spark.SparkConf >>>>>> import org.apache.spark.sql.SQLContext >>>>>> object Sequence { >>>>>> def main(args: Array[String]) { >>>>>> val conf = new SparkConf().setAppName("Sequence") >>>>>> val sc = new SparkContext(conf) >>>>>> val sqlContext = new org.apache.spark.sql.SQLContext(sc) >>>>>> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) >>>>>> val a = Seq(("Mich",20), ("Christian", 18), ("James",13), >>>>>> ("Richard",16)) >>>>>> // Sort option 1 using tempTable >>>>>> val b = a.toDF("Name","score").registerTempTable("tmp") >>>>>> sql("select Name,score from tmp order by score desc").show >>>>>> // Sort option 2 with FP >>>>>> a.toDF("Name","score").sort(desc("score")).show >>>>>> } >>>>>> } >>>>>> >>>>>> I build this using sbt tool as below >>>>>> >>>>>> cat sequence.sbt >>>>>> name := "Sequence" >>>>>> version := "1.0" >>>>>> scalaVersion := "2.10.5" >>>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0" >>>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0" >>>>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.0" >>>>>> >>>>>> >>>>>> But it fails compilation as below >>>>>> >>>>>> [info] Compilation completed in 12.366 s >>>>>> [error] >>>>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: >>>>>> value >>>>>> toDF is not a member of Seq[(String, Int)] >>>>>> [error] val b = a.toDF("Name","score").registerTempTable("tmp") >>>>>> [error] ^ >>>>>> [error] >>>>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: not >>>>>> found: value sql >>>>>> [error] sql("select Name,score from tmp order by score desc").show >>>>>> [error] ^ >>>>>> [error] >>>>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:18: >>>>>> value >>>>>> toDF is not a member of Seq[(String, Int)] >>>>>> [error] a.toDF("Name","score").sort(desc("score")).show >>>>>> [error] ^ >>>>>> [error] three errors found >>>>>> [error] (compile:compileIncremental) Compilation failed >>>>>> [error] Total time: 95 s, completed Mar 4, 2016 9:06:40 PM >>>>>> >>>>>> I think I am missing some dependencies here >>>>>> >>>>>> >>>>>> I have a simple >>>>>> Dr Mich Talebzadeh >>>>>> >>>>>> >>>>>> LinkedIn * >>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>> >>>>>> >>>>>> http://talebzadehmich.wordpress.com >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> >> >