`SQLContext.getOrCreate` will return the HiveContext you created.
On Mon, Nov 14, 2016 at 11:17 PM, Praseetha <prasikris...@gmail.com> wrote:
>
> Hi All,
>
>
> I have a streaming app and when i try invoking the
> HiveContext.getOrCreate, it errors out with the following stmt. 'object
> HiveContext in package hive cannot be accessed in package
> org.apache.spark.sql.hive'
>
> I would require HiveContext instead of SQLContext for my application and
> creating new HiveContext everytime would not be a feasible solution.
>
> Here is my code snippet:
> object sampleStreamingApp {
>
> def createStreamingContext(checkpointDirectory: String):
> StreamingContext = {
> val conf = new SparkConf().setAppName("sampleStreaming")
> val sc = new SparkContext(conf)
> val ssc = new StreamingContext(sc, Milliseconds(5000))
> ssc.checkpoint(checkpointDirectory)
> val smDStream = ssc.textFileStream("/user/hdpuser/data")
> val smSplitted = smDStream.map( x => x.split(";") ).map( x => Row.fromSeq(
> x ) )
> smSplitted.foreachRDD { rdd =>
> val sqlContext = HiveContext.getOrCreate(rdd.
> sparkContext)
> import sqlContext.implicits._
> <other code logic goes here>
> }
> }
> ssc
> }
>
> def main(args: Array[String]) {
> val checkpointDirectory = "hdfs://localhost:8020/user/
> dfml/checkpointing/AAA"
> val ssc = StreamingContext.getActiveOrCreate(checkpointDirectory, () =>
> createStreamingContext(checkpointDirectory))
>
> ssc.start()
> ssc.awaitTermination()
> }
> }
>
> Any help would be appreciated.
>
> Regds,
> --Praseetha
>
