Hi I am using a shared sparkContext for all of my Spark jobs. Some of the jobs use HiveContext, but there isn't a getOrCreate method on HiveContext which will allow reuse of an existing HiveContext. Such a method exists on SQLContext only (def getOrCreate(sparkContext: SparkContext): SQLContext).
Is there any reason that a HiveContext cannot be shared amongst multiple threads within the same Spark driver process? In addition I cannot seem to be able to cast a HiveContext to a SQLContext, but this works fine in the spark shell, I am doing something wrong here? scala> sqlContext res19: org.apache.spark.sql.SQLContext = org.apache.spark.sql.hive.HiveContext@383b3357 scala> import org.apache.spark.sql.SQLContext import org.apache.spark.sql.SQLContext scala> SQLContext.getOrCreate(sc) res18: org.apache.spark.sql.SQLContext = org.apache.spark.sql.hive.HiveContext@383b3357 Regards Deenar
