Can a tempTable registered by sqlContext be used inside a forEachRDD?

[SRK]

Hi,

Can a tempTable registered in sqlContext be used to query inside forEachRDD
as shown below?
My requirement is that I have a set of data in the form of parquet inside
hdfs and I need to register the data
as a tempTable using sqlContext and query it inside forEachRDD as shown
below. 

  sqlContext.registerTempTable("tempTable")

messages.foreachRDD { rdd =>
  val message:RDD[String] = rdd.map { y => y._2 }

  sqlContext.sql("SELECT time,To FROM tempTable")
}

Thanks!



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Can-a-tempTable-registered-by-sqlContext-be-used-inside-a-forEachRDD-tp25862.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Can a tempTable registered by sqlContext be used inside a forEachRDD?

[Sathish Kumaran Vairavelu]

I think you can use foreachpartition instead of foreachrdd


Sathish
On Sun, Jan 3, 2016 at 5:51 AM SRK  wrote:

> Hi,
>
> Can a tempTable registered in sqlContext be used to query inside forEachRDD
> as shown below?
> My requirement is that I have a set of data in the form of parquet inside
> hdfs and I need to register the data
> as a tempTable using sqlContext and query it inside forEachRDD as shown
> below.
>
>   sqlContext.registerTempTable("tempTable")
>
> messages.foreachRDD { rdd =>
>   val message:RDD[String] = rdd.map { y => y._2 }
>
>   sqlContext.sql("SELECT time,To FROM tempTable")
> }
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Can-a-tempTable-registered-by-sqlContext-be-used-inside-a-forEachRDD-tp25862.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>