subject:"Re\: Error building a self contained Spark app"

Re: Error building a self contained Spark app

2016-03-04 Thread Chandeep Singh

#3 If your code is dependent on other projects you will need to package 
everything together in order to distribute over a Spark cluster.

In your example below I don’t see much of an advantage by building a package.

> On Mar 5, 2016, at 12:32 AM, Ted Yu  wrote:
> 
> Answers to first two questions are 'yes'
> 
> Not clear on what the 3rd question is asking.
> 
> On Fri, Mar 4, 2016 at 4:28 PM, Mich Talebzadeh  > wrote:
> Thanks now all working. Also select from  tmp tables are part of sqlContext 
> not HiveContext
> 
> This is the final code that works in blue
> 
> 
> Couple of questions if I may
> 
> This works pretty effortless in spark-shell. Is this because $CLASSPATH 
> already includes all the needed jars?
> The import section. That imports the needed classes. So basically import 
> org.apache.spark.sql.functions._ imports all the methods of Class functions?
> What is the reason why we should use sbt to build custom jars from a spark 
> code as opposed to running the code against spark shell in a file? Any 
> particular use case for it?
> 
> import org.apache.spark.SparkContext
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.hive.HiveContext
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.SQLContext
> import org.apache.spark.sql.functions._
> //
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new 
> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
>  "true")
>   val sc = new SparkContext(conf)
>   // Note that this should be done only after an instance of 
> org.apache.spark.sql.SQLContext is created. It should be written as:
>   val sqlContext= new org.apache.spark.sql.SQLContext(sc)
>   import sqlContext.implicits._
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   sqlContext.sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
>  
> 
> On 4 March 2016 at 23:58, Chandeep Singh  > wrote:
> That is because an instance of org.apache.spark.sql.SQLContext doesn’t exist 
> in the current context and is required before you can use any of its implicit 
> methods.
> 
> As Ted mentioned importing org.apache.spark.sql.functions._ will take care of 
> the below error.
> 
> 
>> On Mar 4, 2016, at 11:35 PM, Mich Talebzadeh > > wrote:
>> 
>> thanks. It is like war of attrition. I always thought that you add  import 
>> before the class itself not within the class? w3hat is the reason for it 
>> please?
>> 
>> this is my code
>> 
>> import org.apache.spark.SparkContext
>> import org.apache.spark.SparkConf
>> import org.apache.spark.sql.Row
>> import org.apache.spark.sql.hive.HiveContext
>> import org.apache.spark.sql.types._
>> import org.apache.spark.sql.SQLContext
>> //
>> object Sequence {
>>   def main(args: Array[String]) {
>>   val conf = new 
>> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
>>  "true")
>>   val sc = new SparkContext(conf)
>>   // Note that this should be done only after an instance of 
>> org.apache.spark.sql.SQLContext is created. It should be written as:
>>   val sqlContext= new org.apache.spark.sql.SQLContext(sc)
>>   import sqlContext.implicits._
>>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>>   // Sort option 1 using tempTable
>>   val b = a.toDF("Name","score").registerTempTable("tmp")
>>   HiveContext.sql("select Name,score from tmp order by score desc").show
>>   // Sort option 2 with FP
>>   a.toDF("Name","score").sort(desc("score")).show
>>  }
>> }
>> 
>> And now the last failure is in
>> 
>> info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (104ms)
>> [info] Done updating.
>> [info] Compiling 1 Scala source to 
>> /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
>> [info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
>> [info]   Compilation completed in 15.779 s
>> [error] 
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:21: not 
>> found: value desc
>> [error]   a.toDF("Name","score").sort(desc("score")).show
>> [error]

Re: Error building a self contained Spark app

2016-03-04 Thread Chandeep Singh

That is because an instance of org.apache.spark.sql.SQLContext doesn’t exist in 
the current context and is required before you can use any of its implicit 
methods.

As Ted mentioned importing org.apache.spark.sql.functions._ will take care of 
the below error.

> On Mar 4, 2016, at 11:35 PM, Mich Talebzadeh  
> wrote:
> 
> thanks. It is like war of attrition. I always thought that you add  import 
> before the class itself not within the class? w3hat is the reason for it 
> please?
> 
> this is my code
> 
> import org.apache.spark.SparkContext
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.hive.HiveContext
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.SQLContext
> //
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new 
> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
>  "true")
>   val sc = new SparkContext(conf)
>   // Note that this should be done only after an instance of 
> org.apache.spark.sql.SQLContext is created. It should be written as:
>   val sqlContext= new org.apache.spark.sql.SQLContext(sc)
>   import sqlContext.implicits._
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   HiveContext.sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
> 
> And now the last failure is in
> 
> info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (104ms)
> [info] Done updating.
> [info] Compiling 1 Scala source to 
> /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
> [info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
> [info]   Compilation completed in 15.779 s
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:21: 
> not found: value desc
> [error]   a.toDF("Name","score").sort(desc("score")).show
> [error]   ^
> [error] one error found
> [error] (compile:compileIncremental) Compilation failed
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
>  
> 
> On 4 March 2016 at 23:25, Chandeep Singh  > wrote:
> This is what you need:
> 
> val sc = new SparkContext(sparkConf)
> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
> import sqlContext.implicits._
> 
>> On Mar 4, 2016, at 11:03 PM, Mich Talebzadeh > > wrote:
>> 
>> Hi Ted,
>> 
>> This is my code
>> 
>> import org.apache.spark.SparkConf
>> import org.apache.spark.sql.Row
>> import org.apache.spark.sql.hive.HiveContext
>> import org.apache.spark.sql.types._
>> import org.apache.spark.sql.SQLContext
>> //
>> object Sequence {
>>   def main(args: Array[String]) {
>>   val conf = new 
>> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
>>  "true")
>>   val sc = new SparkContext(conf)
>>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>>   // Sort option 1 using tempTable
>>   val b = a.toDF("Name","score").registerTempTable("tmp")
>>   sql("select Name,score from tmp order by score desc").show
>>   // Sort option 2 with FP
>>   a.toDF("Name","score").sort(desc("score")).show
>>  }
>> }
>> 
>> And the error I am getting now is
>> 
>> [info] downloading 
>> https://repo1.maven.org/maven2/org/scala-lang/jline/2.10.5/jline-2.10.5.jar 
>> 
>>  ...
>> [info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (103ms)
>> [info] Done updating.
>> [info] Compiling 1 Scala source to 
>> /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
>> [info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
>> [info]   Compilation completed in 12.462 s
>> [error] 
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: value 
>> toDF is not a member of Seq[(String, Int)]
>> [error]   val b = a.toDF("Name","score").registerTempTable("tmp")
>> [error] ^
>> [error] 
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:17: not 
>> found: value sql
>> [error]   sql("select Name,score from tmp order by score desc").show
>> [error]   ^
>> [error] 
>>

Re: Error building a self contained Spark app

2016-03-04 Thread Ted Yu

Please import:

import org.apache.spark.sql.functions._

On Fri, Mar 4, 2016 at 3:35 PM, Mich Talebzadeh 
wrote:

> thanks. It is like war of attrition. I always thought that you add  import
> before the class itself not within the class? w3hat is the reason for it
> please?
>
> this is my code
>
> import org.apache.spark.SparkContext
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.hive.HiveContext
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.SQLContext
> //
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new
> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
> "true")
>   val sc = new SparkContext(conf)
>   // Note that this should be done only after an instance of
> org.apache.spark.sql.SQLContext is created. It should be written as:
>   val sqlContext= new org.apache.spark.sql.SQLContext(sc)
>   import sqlContext.implicits._
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   HiveContext.sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
>
> And now the last failure is in
>
> info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (104ms)
> [info] Done updating.
> [info] Compiling 1 Scala source to
> /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
> [info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
> [info]   Compilation completed in 15.779 s
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:21: not
> found: value desc
> [error]   a.toDF("Name","score").sort(desc("score")).show
> [error]   ^
> [error] one error found
> [error] (compile:compileIncremental) Compilation failed
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 4 March 2016 at 23:25, Chandeep Singh  wrote:
>
>> This is what you need:
>>
>> val sc = new SparkContext(sparkConf)
>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>> import sqlContext.implicits._
>>
>> On Mar 4, 2016, at 11:03 PM, Mich Talebzadeh 
>> wrote:
>>
>> Hi Ted,
>>
>> This is my code
>>
>> import org.apache.spark.SparkConf
>> import org.apache.spark.sql.Row
>> import org.apache.spark.sql.hive.HiveContext
>> import org.apache.spark.sql.types._
>> import org.apache.spark.sql.SQLContext
>> //
>> object Sequence {
>>   def main(args: Array[String]) {
>>   val conf = new
>> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
>> "true")
>>   val sc = new SparkContext(conf)
>>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13),
>> ("Richard",16))
>>   // Sort option 1 using tempTable
>>   val b = a.toDF("Name","score").registerTempTable("tmp")
>>   sql("select Name,score from tmp order by score desc").show
>>   // Sort option 2 with FP
>>   a.toDF("Name","score").sort(desc("score")).show
>>  }
>> }
>>
>> And the error I am getting now is
>>
>> [info] downloading
>> https://repo1.maven.org/maven2/org/scala-lang/jline/2.10.5/jline-2.10.5.jar
>> ...
>> [info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (103ms)
>> [info] Done updating.
>> [info] Compiling 1 Scala source to
>> /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
>> [info] 'compiler-interface' not yet compiled for Scala 2.10.5.
>> Compiling...
>> [info]   Compilation completed in 12.462 s
>> [error]
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: value
>> toDF is not a member of Seq[(String, Int)]
>> [error]   val b = a.toDF("Name","score").registerTempTable("tmp")
>> [error] ^
>> [error]
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:17: not
>> found: value sql
>> [error]   sql("select Name,score from tmp order by score desc").show
>> [error]   ^
>> [error]
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:19: value
>> toDF is not a member of Seq[(String, Int)]
>> [error]   a.toDF("Name","score").sort(desc("score")).show
>> [error] ^
>> [error] three errors found
>> [error] (compile:compileIncremental) Compilation failed
>> [error] Total time: 88 s, completed Mar 4, 2016 11:12:46 PM
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>> LinkedIn * 
>>

Re: Error building a self contained Spark app

2016-03-04 Thread Mich Talebzadeh

thanks. It is like war of attrition. I always thought that you add  import
before the class itself not within the class? w3hat is the reason for it
please?

this is my code

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.Row
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.types._
import org.apache.spark.sql.SQLContext
//
object Sequence {
  def main(args: Array[String]) {
  val conf = new
SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
"true")
  val sc = new SparkContext(conf)
  // Note that this should be done only after an instance of
org.apache.spark.sql.SQLContext is created. It should be written as:
  val sqlContext= new org.apache.spark.sql.SQLContext(sc)
  import sqlContext.implicits._
  val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
  val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
  // Sort option 1 using tempTable
  val b = a.toDF("Name","score").registerTempTable("tmp")
  HiveContext.sql("select Name,score from tmp order by score desc").show
  // Sort option 2 with FP
  a.toDF("Name","score").sort(desc("score")).show
 }
}

And now the last failure is in

info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (104ms)
[info] Done updating.
[info] Compiling 1 Scala source to
/home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
[info]   Compilation completed in 15.779 s
[error]
/home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:21: not
found: value desc
[error]   a.toDF("Name","score").sort(desc("score")).show
[error]   ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed

Dr Mich Talebzadeh

LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*

http://talebzadehmich.wordpress.com

On 4 March 2016 at 23:25, Chandeep Singh  wrote:

> This is what you need:
>
> val sc = new SparkContext(sparkConf)
> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
> import sqlContext.implicits._
>
> On Mar 4, 2016, at 11:03 PM, Mich Talebzadeh 
> wrote:
>
> Hi Ted,
>
> This is my code
>
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.hive.HiveContext
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.SQLContext
> //
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new
> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
> "true")
>   val sc = new SparkContext(conf)
>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
>
> And the error I am getting now is
>
> [info] downloading
> https://repo1.maven.org/maven2/org/scala-lang/jline/2.10.5/jline-2.10.5.jar
> ...
> [info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (103ms)
> [info] Done updating.
> [info] Compiling 1 Scala source to
> /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
> [info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
> [info]   Compilation completed in 12.462 s
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: value
> toDF is not a member of Seq[(String, Int)]
> [error]   val b = a.toDF("Name","score").registerTempTable("tmp")
> [error] ^
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:17: not
> found: value sql
> [error]   sql("select Name,score from tmp order by score desc").show
> [error]   ^
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:19: value
> toDF is not a member of Seq[(String, Int)]
> [error]   a.toDF("Name","score").sort(desc("score")).show
> [error] ^
> [error] three errors found
> [error] (compile:compileIncremental) Compilation failed
> [error] Total time: 88 s, completed Mar 4, 2016 11:12:46 PM
>
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 4 March 2016 at 22:52, Ted Yu  wrote:
>
>> Can you show your code snippet ?
>> Here is an example:
>>
>>   val sqlContext = new SQLContext(sc)
>>

Re: Error building a self contained Spark app

2016-03-04 Thread Chandeep Singh

This is what you need:

val sc = new SparkContext(sparkConf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._

> On Mar 4, 2016, at 11:03 PM, Mich Talebzadeh  
> wrote:
> 
> Hi Ted,
> 
> This is my code
> 
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.hive.HiveContext
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.SQLContext
> //
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new 
> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
>  "true")
>   val sc = new SparkContext(conf)
>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
> 
> And the error I am getting now is
> 
> [info] downloading 
> https://repo1.maven.org/maven2/org/scala-lang/jline/2.10.5/jline-2.10.5.jar 
>  
> ...
> [info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (103ms)
> [info] Done updating.
> [info] Compiling 1 Scala source to 
> /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
> [info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
> [info]   Compilation completed in 12.462 s
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: 
> value toDF is not a member of Seq[(String, Int)]
> [error]   val b = a.toDF("Name","score").registerTempTable("tmp")
> [error] ^
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:17: 
> not found: value sql
> [error]   sql("select Name,score from tmp order by score desc").show
> [error]   ^
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:19: 
> value toDF is not a member of Seq[(String, Int)]
> [error]   a.toDF("Name","score").sort(desc("score")).show
> [error] ^
> [error] three errors found
> [error] (compile:compileIncremental) Compilation failed
> [error] Total time: 88 s, completed Mar 4, 2016 11:12:46 PM
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
>  
> 
> On 4 March 2016 at 22:52, Ted Yu  > wrote:
> Can you show your code snippet ?
> Here is an example:
> 
>   val sqlContext = new SQLContext(sc)
>   import sqlContext.implicits._
> 
> On Fri, Mar 4, 2016 at 1:55 PM, Mich Talebzadeh  > wrote:
> Hi Ted,
> 
>  I am getting the following error after adding that import
> 
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:5: 
> not found: object sqlContext
> [error] import sqlContext.implicits._
> [error]^
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: 
> value toDF is not a member of Seq[(String, Int)]
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
>  
> 
> On 4 March 2016 at 21:39, Ted Yu  > wrote:
> Can you add the following into your code ?
>  import sqlContext.implicits._
> 
> On Fri, Mar 4, 2016 at 1:14 PM, Mich Talebzadeh  > wrote:
> Hi,
> 
> I have a simple Scala program as below
> 
> import org.apache.spark.SparkContext
> import org.apache.spark.SparkContext._
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.SQLContext
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new SparkConf().setAppName("Sequence")
>   val sc = new SparkContext(conf)
>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
> 
> I build this using sbt tool as below
> 
>

Re: Error building a self contained Spark app

2016-03-04 Thread Ted Yu

After:
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)
Please add:
  import sqlContext.implicits._

On Fri, Mar 4, 2016 at 3:03 PM, Mich Talebzadeh 
wrote:

> Hi Ted,
>
> This is my code
>
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.hive.HiveContext
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.SQLContext
> //
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new
> SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
> "true")
>   val sc = new SparkContext(conf)
>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
>
> And the error I am getting now is
>
> [info] downloading
> https://repo1.maven.org/maven2/org/scala-lang/jline/2.10.5/jline-2.10.5.jar
> ...
> [info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (103ms)
> [info] Done updating.
> [info] Compiling 1 Scala source to
> /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
> [info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
> [info]   Compilation completed in 12.462 s
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: value
> toDF is not a member of Seq[(String, Int)]
> [error]   val b = a.toDF("Name","score").registerTempTable("tmp")
> [error] ^
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:17: not
> found: value sql
> [error]   sql("select Name,score from tmp order by score desc").show
> [error]   ^
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:19: value
> toDF is not a member of Seq[(String, Int)]
> [error]   a.toDF("Name","score").sort(desc("score")).show
> [error] ^
> [error] three errors found
> [error] (compile:compileIncremental) Compilation failed
> [error] Total time: 88 s, completed Mar 4, 2016 11:12:46 PM
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 4 March 2016 at 22:52, Ted Yu  wrote:
>
>> Can you show your code snippet ?
>> Here is an example:
>>
>>   val sqlContext = new SQLContext(sc)
>>   import sqlContext.implicits._
>>
>> On Fri, Mar 4, 2016 at 1:55 PM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> Hi Ted,
>>>
>>>  I am getting the following error after adding that import
>>>
>>> [error]
>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:5: not
>>> found: object sqlContext
>>> [error] import sqlContext.implicits._
>>> [error]^
>>> [error]
>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value
>>> toDF is not a member of Seq[(String, Int)]
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 4 March 2016 at 21:39, Ted Yu  wrote:
>>>
 Can you add the following into your code ?
  import sqlContext.implicits._

 On Fri, Mar 4, 2016 at 1:14 PM, Mich Talebzadeh <
 mich.talebza...@gmail.com> wrote:

> Hi,
>
> I have a simple Scala program as below
>
> import org.apache.spark.SparkContext
> import org.apache.spark.SparkContext._
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.SQLContext
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new SparkConf().setAppName("Sequence")
>   val sc = new SparkContext(conf)
>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13),
> ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
>
> I build this using sbt tool as below
>
>  cat sequence.sbt
> name := "Sequence"
> version := "1.0"
> scalaVersion := "2.10.5"
> libraryDependencies += "org.apache.spark" %%

Re: Error building a self contained Spark app

2016-03-04 Thread Mich Talebzadeh

Hi Ted,

This is my code

import org.apache.spark.SparkConf
import org.apache.spark.sql.Row
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.types._
import org.apache.spark.sql.SQLContext
//
object Sequence {
  def main(args: Array[String]) {
  val conf = new
SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
"true")
  val sc = new SparkContext(conf)
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
  val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
  // Sort option 1 using tempTable
  val b = a.toDF("Name","score").registerTempTable("tmp")
  sql("select Name,score from tmp order by score desc").show
  // Sort option 2 with FP
  a.toDF("Name","score").sort(desc("score")).show
 }
}

And the error I am getting now is

[info] downloading
https://repo1.maven.org/maven2/org/scala-lang/jline/2.10.5/jline-2.10.5.jar
...
[info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (103ms)
[info] Done updating.
[info] Compiling 1 Scala source to
/home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
[info]   Compilation completed in 12.462 s
[error]
/home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: value
toDF is not a member of Seq[(String, Int)]
[error]   val b = a.toDF("Name","score").registerTempTable("tmp")
[error] ^
[error]
/home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:17: not
found: value sql
[error]   sql("select Name,score from tmp order by score desc").show
[error]   ^
[error]
/home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:19: value
toDF is not a member of Seq[(String, Int)]
[error]   a.toDF("Name","score").sort(desc("score")).show
[error] ^
[error] three errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 88 s, completed Mar 4, 2016 11:12:46 PM


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 4 March 2016 at 22:52, Ted Yu  wrote:

> Can you show your code snippet ?
> Here is an example:
>
>   val sqlContext = new SQLContext(sc)
>   import sqlContext.implicits._
>
> On Fri, Mar 4, 2016 at 1:55 PM, Mich Talebzadeh  > wrote:
>
>> Hi Ted,
>>
>>  I am getting the following error after adding that import
>>
>> [error]
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:5: not
>> found: object sqlContext
>> [error] import sqlContext.implicits._
>> [error]^
>> [error]
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value
>> toDF is not a member of Seq[(String, Int)]
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 4 March 2016 at 21:39, Ted Yu  wrote:
>>
>>> Can you add the following into your code ?
>>>  import sqlContext.implicits._
>>>
>>> On Fri, Mar 4, 2016 at 1:14 PM, Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 Hi,

 I have a simple Scala program as below

 import org.apache.spark.SparkContext
 import org.apache.spark.SparkContext._
 import org.apache.spark.SparkConf
 import org.apache.spark.sql.SQLContext
 object Sequence {
   def main(args: Array[String]) {
   val conf = new SparkConf().setAppName("Sequence")
   val sc = new SparkContext(conf)
   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
   val a = Seq(("Mich",20), ("Christian", 18), ("James",13),
 ("Richard",16))
   // Sort option 1 using tempTable
   val b = a.toDF("Name","score").registerTempTable("tmp")
   sql("select Name,score from tmp order by score desc").show
   // Sort option 2 with FP
   a.toDF("Name","score").sort(desc("score")).show
  }
 }

 I build this using sbt tool as below

  cat sequence.sbt
 name := "Sequence"
 version := "1.0"
 scalaVersion := "2.10.5"
 libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0"
 libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0"
 libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.0"


 But it fails compilation as below

 [info]   Compilation completed in 12.366 s
 [error]
 /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value
 toDF is not a member of Seq[(String, Int)]

RE: Error building a self contained Spark app

2016-03-04 Thread Jelez Raditchkov

Ok this is what I have:
 
object SQLHiveContextSingleton {
@transient private var instance: HiveContext = _
def getInstance(sparkContext: SparkContext): HiveContext = {
synchronized {
if (instance == null || sparkContext.isStopped) {
instance = new HiveContext(sparkContext)
}
instance
}
}
}
and then in my application I have:
linesStream.foreachRDD(linesRdd => {
 if (!linesRdd.isEmpty()) {
  val jsonRdd = linesRdd.map(x => parser.parse(x))
  val sqlContext = SQLHiveContextSingleton.getInstance(linesRdd.sparkContext)
  import sqlContext.implicits._
  
Again this is for a streaming app and I am following best practice from here: 
https://spark.apache.org/docs/latest/streaming-programming-guide.html#dataframe-and-sql-operations
 
Date: Fri, 4 Mar 2016 14:52:58 -0800
Subject: Re: Error building a self contained Spark app
From: yuzhih...@gmail.com
To: mich.talebza...@gmail.com
CC: user@spark.apache.org

Can you show your code snippet ?Here is an example:
  val sqlContext = new SQLContext(sc)  import sqlContext.implicits._
On Fri, Mar 4, 2016 at 1:55 PM, Mich Talebzadeh <mich.talebza...@gmail.com> 
wrote:
Hi Ted,
 I am getting the following error after adding that import
[error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:5: 
not found: object sqlContext
[error] import sqlContext.implicits._
[error]^
[error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: 
value toDF is not a member of Seq[(String, Int)]


Dr Mich Talebzadeh


 


LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw


 


http://talebzadehmich.wordpress.com

 




On 4 March 2016 at 21:39, Ted Yu <yuzhih...@gmail.com> wrote:
Can you add the following into your code ? import sqlContext.implicits._

On Fri, Mar 4, 2016 at 1:14 PM, Mich Talebzadeh <mich.talebza...@gmail.com> 
wrote:
Hi,
I have a simple Scala program as below
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContextobject Sequence {
  def main(args: Array[String]) {
  val conf = new SparkConf().setAppName("Sequence")
  val sc = new SparkContext(conf)
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
  val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
  // Sort option 1 using tempTable
  val b = a.toDF("Name","score").registerTempTable("tmp")
  sql("select Name,score from tmp order by score desc").show
  // Sort option 2 with FP
  a.toDF("Name","score").sort(desc("score")).show
 }
}
I build this using sbt tool as below
 cat sequence.sbt
name := "Sequence"version := "1.0"scalaVersion := "2.10.5"libraryDependencies 
+= "org.apache.spark" %% "spark-core" % "1.5.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.0"

But it fails compilation as below
[info]   Compilation completed in 12.366 s
[error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: 
value toDF is not a member of Seq[(String, Int)]
[error]   val b = a.toDF("Name","score").registerTempTable("tmp")
[error] ^
[error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: 
not found: value sql
[error]   sql("select Name,score from tmp order by score desc").show
[error]   ^
[error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:18: 
value toDF is not a member of Seq[(String, Int)]
[error]   a.toDF("Name","score").sort(desc("score")).show
[error] ^
[error] three errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 95 s, completed Mar 4, 2016 9:06:40 PM

I think I am missing some dependencies here

I have a simple 

Dr Mich Talebzadeh


 


LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw


 


http://talebzadehmich.wordpress.com

Re: Error building a self contained Spark app

2016-03-04 Thread Ted Yu

Can you show your code snippet ?
Here is an example:

  val sqlContext = new SQLContext(sc)
  import sqlContext.implicits._

On Fri, Mar 4, 2016 at 1:55 PM, Mich Talebzadeh 
wrote:

> Hi Ted,
>
>  I am getting the following error after adding that import
>
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:5: not
> found: object sqlContext
> [error] import sqlContext.implicits._
> [error]^
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value
> toDF is not a member of Seq[(String, Int)]
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 4 March 2016 at 21:39, Ted Yu  wrote:
>
>> Can you add the following into your code ?
>>  import sqlContext.implicits._
>>
>> On Fri, Mar 4, 2016 at 1:14 PM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a simple Scala program as below
>>>
>>> import org.apache.spark.SparkContext
>>> import org.apache.spark.SparkContext._
>>> import org.apache.spark.SparkConf
>>> import org.apache.spark.sql.SQLContext
>>> object Sequence {
>>>   def main(args: Array[String]) {
>>>   val conf = new SparkConf().setAppName("Sequence")
>>>   val sc = new SparkContext(conf)
>>>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>>>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>>>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13),
>>> ("Richard",16))
>>>   // Sort option 1 using tempTable
>>>   val b = a.toDF("Name","score").registerTempTable("tmp")
>>>   sql("select Name,score from tmp order by score desc").show
>>>   // Sort option 2 with FP
>>>   a.toDF("Name","score").sort(desc("score")).show
>>>  }
>>> }
>>>
>>> I build this using sbt tool as below
>>>
>>>  cat sequence.sbt
>>> name := "Sequence"
>>> version := "1.0"
>>> scalaVersion := "2.10.5"
>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0"
>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0"
>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.0"
>>>
>>>
>>> But it fails compilation as below
>>>
>>> [info]   Compilation completed in 12.366 s
>>> [error]
>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value
>>> toDF is not a member of Seq[(String, Int)]
>>> [error]   val b = a.toDF("Name","score").registerTempTable("tmp")
>>> [error] ^
>>> [error]
>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: not
>>> found: value sql
>>> [error]   sql("select Name,score from tmp order by score desc").show
>>> [error]   ^
>>> [error]
>>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:18: value
>>> toDF is not a member of Seq[(String, Int)]
>>> [error]   a.toDF("Name","score").sort(desc("score")).show
>>> [error] ^
>>> [error] three errors found
>>> [error] (compile:compileIncremental) Compilation failed
>>> [error] Total time: 95 s, completed Mar 4, 2016 9:06:40 PM
>>>
>>> I think I am missing some dependencies here
>>>
>>>
>>> I have a simple
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>
>>
>

Re: Error building a self contained Spark app

2016-03-04 Thread Mich Talebzadeh

Hi Ted,

 I am getting the following error after adding that import

[error]
/home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:5: not
found: object sqlContext
[error] import sqlContext.implicits._
[error]^
[error]
/home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value
toDF is not a member of Seq[(String, Int)]


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 4 March 2016 at 21:39, Ted Yu  wrote:

> Can you add the following into your code ?
>  import sqlContext.implicits._
>
> On Fri, Mar 4, 2016 at 1:14 PM, Mich Talebzadeh  > wrote:
>
>> Hi,
>>
>> I have a simple Scala program as below
>>
>> import org.apache.spark.SparkContext
>> import org.apache.spark.SparkContext._
>> import org.apache.spark.SparkConf
>> import org.apache.spark.sql.SQLContext
>> object Sequence {
>>   def main(args: Array[String]) {
>>   val conf = new SparkConf().setAppName("Sequence")
>>   val sc = new SparkContext(conf)
>>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13),
>> ("Richard",16))
>>   // Sort option 1 using tempTable
>>   val b = a.toDF("Name","score").registerTempTable("tmp")
>>   sql("select Name,score from tmp order by score desc").show
>>   // Sort option 2 with FP
>>   a.toDF("Name","score").sort(desc("score")).show
>>  }
>> }
>>
>> I build this using sbt tool as below
>>
>>  cat sequence.sbt
>> name := "Sequence"
>> version := "1.0"
>> scalaVersion := "2.10.5"
>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0"
>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0"
>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.0"
>>
>>
>> But it fails compilation as below
>>
>> [info]   Compilation completed in 12.366 s
>> [error]
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value
>> toDF is not a member of Seq[(String, Int)]
>> [error]   val b = a.toDF("Name","score").registerTempTable("tmp")
>> [error] ^
>> [error]
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: not
>> found: value sql
>> [error]   sql("select Name,score from tmp order by score desc").show
>> [error]   ^
>> [error]
>> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:18: value
>> toDF is not a member of Seq[(String, Int)]
>> [error]   a.toDF("Name","score").sort(desc("score")).show
>> [error] ^
>> [error] three errors found
>> [error] (compile:compileIncremental) Compilation failed
>> [error] Total time: 95 s, completed Mar 4, 2016 9:06:40 PM
>>
>> I think I am missing some dependencies here
>>
>>
>> I have a simple
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>
>

Re: Error building a self contained Spark app

2016-03-04 Thread Ted Yu

Can you add the following into your code ?
 import sqlContext.implicits._

On Fri, Mar 4, 2016 at 1:14 PM, Mich Talebzadeh 
wrote:

> Hi,
>
> I have a simple Scala program as below
>
> import org.apache.spark.SparkContext
> import org.apache.spark.SparkContext._
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.SQLContext
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new SparkConf().setAppName("Sequence")
>   val sc = new SparkContext(conf)
>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
>
> I build this using sbt tool as below
>
>  cat sequence.sbt
> name := "Sequence"
> version := "1.0"
> scalaVersion := "2.10.5"
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0"
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0"
> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.0"
>
>
> But it fails compilation as below
>
> [info]   Compilation completed in 12.366 s
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value
> toDF is not a member of Seq[(String, Int)]
> [error]   val b = a.toDF("Name","score").registerTempTable("tmp")
> [error] ^
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: not
> found: value sql
> [error]   sql("select Name,score from tmp order by score desc").show
> [error]   ^
> [error]
> /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:18: value
> toDF is not a member of Seq[(String, Int)]
> [error]   a.toDF("Name","score").sort(desc("score")).show
> [error] ^
> [error] three errors found
> [error] (compile:compileIncremental) Compilation failed
> [error] Total time: 95 s, completed Mar 4, 2016 9:06:40 PM
>
> I think I am missing some dependencies here
>
>
> I have a simple
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>

Re: Error building a self contained Spark app

Re: Error building a self contained Spark app

Re: Error building a self contained Spark app

Re: Error building a self contained Spark app

Re: Error building a self contained Spark app

Re: Error building a self contained Spark app

Re: Error building a self contained Spark app

RE: Error building a self contained Spark app

Re: Error building a self contained Spark app

Re: Error building a self contained Spark app

Re: Error building a self contained Spark app

11 matches

Site Navigation

Mail list logo

Footer information