Re: [Spark-SQL]: Disable HiveContext from instantiating in spark-shell

Hi Zhan, Thank you for providing a workaround! I will try this out but I agree with Ted, there should be a better way to capture the exception and handle it by just initializing SQLContext instead of HiveContext. WARN the user that something is wrong with his hive setup. Having

Re: [Spark-SQL]: Disable HiveContext from instantiating in spark-shell

Hi Jerry, OK. Here is an ugly walk around. Put a hive-site.xml under $SPARK_HOME/conf with invalid content. You will get a bunch of exceptions because hive context initialization failure, but you can initialize your SQLContext on your own. scala> val sqlContext = new

Re: [Spark-SQL]: Disable HiveContext from instantiating in spark-shell

I agree with minor change. Adding a config to provide the option to init SQLContext or HiveContext, with HiveContext as default instead of bypassing when hitting the Exception. Thanks. Zhan Zhang On Nov 6, 2015, at 2:53 PM, Ted Yu > wrote: I

Re: [Spark-SQL]: Disable HiveContext from instantiating in spark-shell

I would suggest adding a config parameter that allows bypassing initialization of HiveContext in case of SQLException Cheers On Fri, Nov 6, 2015 at 2:50 PM, Zhan Zhang wrote: > Hi Jerry, > > OK. Here is an ugly walk around. > > Put a hive-site.xml under $SPARK_HOME/conf

Re: [Spark-SQL]: Disable HiveContext from instantiating in spark-shell

If you assembly jar have hive jar included, the HiveContext will be used. Typically, HiveContext has more functionality than SQLContext. In what case you have to use SQLContext that cannot be done by HiveContext? Thanks. Zhan Zhang On Nov 6, 2015, at 10:43 AM, Jerry Lam

Re: [Spark-SQL]: Disable HiveContext from instantiating in spark-shell

Hi Zhan, I don’t use HiveContext features at all. I use mostly DataFrame API. It is sexier and much less typo. :) Also, HiveContext requires metastore database setup (derby by default). The problem is that I cannot have 2 spark-shell sessions running at the same time in the same host (e.g.

Re: [Spark-SQL]: Disable HiveContext from instantiating in spark-shell

Hi Jerry, https://issues.apache.org/jira/browse/SPARK-11562 is created for the issue. Thanks. Zhan Zhang On Nov 6, 2015, at 3:01 PM, Jerry Lam > wrote: Hi Zhan, Thank you for providing a workaround! I will try this out but I agree with Ted,

Re: [Spark-SQL]: Disable HiveContext from instantiating in spark-shell

Hi Ted, I was trying to set spark.sql.dialect to sql as to specify I only need “SQLContext” not HiveContext. It didn’t work. It still instantiate HiveContext. Since I don’t use HiveContext and I don’t want to start a mysql database because I want to have more than 1 session of spark-shell

[Spark-SQL]: Disable HiveContext from instantiating in spark-shell

Hi spark users and developers, Is it possible to disable HiveContext from being instantiated when using spark-shell? I got the following errors when I have more than one session starts. Since I don't use HiveContext, it would be great if I can have more than 1 spark-shell start at the same time.

Re: [Spark-SQL]: Disable HiveContext from instantiating in spark-shell

What is interesting is that pyspark shell works fine with multiple session in the same host even though multiple HiveContext has been created. What does pyspark does differently in terms of starting up the shell? > On Nov 6, 2015, at 12:12 PM, Ted Yu wrote: > > In