Hi Gerald, I am not sure the so called independence is will. I gather you want to use HiveContext for your SQL queries and sqlContext only provides a subset of HiveContext.
try this val sc = new SparkContext(conf) // Create sqlContext based on HiveContext val sqlContext = new HiveContext(sc) However, ii will take 3 minutes to set up hive and all you need to add a softlink from $SPARK_HOME/conf to hive-site.xml hive-site.xml -> /usr/lib/hive/conf/hive-site.xml The fact that it is not working shows that the statement in doc may not be valid. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 26 May 2016 at 10:28, Gerard Maas <gerard.m...@gmail.com> wrote: > Hi, > > I'm helping some folks setting up an analytics cluster with Spark. > They want to use the HiveContext to enable the Window functions on > DataFrames(*) but they don't have any Hive installation, nor they need one > at the moment (if not necessary for this feature) > > When we try to create a Hive context, we get the following error: > > > val sqlContext = new org.apache.spark.sql.hive.HiveContext(sparkContext) > > java.lang.RuntimeException: java.lang.RuntimeException: Unable to > instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) > > Is my HiveContext failing b/c it wants to connect to an unconfigured Hive > Metastore? > > Is there a way to instantiate a HiveContext for the sake of Window > support without an underlying Hive deployment? > > The docs are explicit in saying that that is should be the case: [1] > > "To use a HiveContext, you do not need to have an existing Hive setup, > and all of the data sources available to aSQLContext are still available. > HiveContext is only packaged separately to avoid including all of Hive’s > dependencies in the default Spark build." > > So what is the right way to address this issue? How to instantiate a > HiveContext with spark running on a HDFS cluster without Hive deployed? > > > Thanks a lot! > > -Gerard. > > (*) The need for a HiveContext to use Window functions is pretty obscure. > The only documentation of this seems to be a runtime exception: " > org.apache.spark.sql.AnalysisException: Could not resolve window function > 'max'. Note that, using window functions currently requires a HiveContext;" > > > [1] > http://spark.apache.org/docs/latest/sql-programming-guide.html#getting-started >
