Re: SQLContext Create Table Problem

[Yin Huai]

Can you try to use HiveContext instead of SQLContext? Your query is trying
to create a table and persist the metadata of the table in metastore, which
is only supported by HiveContext.

On Wed, Aug 19, 2015 at 8:44 AM, Yusuf Can Gürkan yu...@useinsider.com
wrote:

 Hello,

 I’m trying to create a table with sqlContext.sql method as below:

 *val sc = new SparkContext()*
 *val sqlContext = new SQLContext(sc)*

 *import sqlContext.implicits._*

 *sqlContext.sql(s*
 *create table if not exists landing (*
 *date string,*
 *referrer string*
 *)*
 *partitioned by (partnerid string,dt string)*
 *row format delimited fields terminated by '\t' lines terminated by '\n'*
 *STORED AS TEXTFILE LOCATION 's3n://...'*
 *  ”)*


 It gives error on spark-submit:

 *Exception in thread main java.lang.RuntimeException: [2.1] failure:
 ``with'' expected but identifier create found*

 *create external table if not exists landing (*

 *^*
 * at scala.sys.package$.error(package.scala:27)*
 * at
 org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)*
 * at
 org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)*



 What can be the reason??

Re: SQLContext Create Table Problem

[Yusuf Can Gürkan]

Hey Yin,

Thanks for answer. I thought that this could be problem but i can not create 
HiveContext because i can not import org.apache.spark.sql.hive.HiveContext. It 
does not see this package. 

I read that i should build spark with -PHive but i’m running on Amazon EMR 
1.4.1 and on spark-shell i can import hive package but can not do the same on 
spark-submit. Do you have any idea why? Because if it’s related to build with 
-PHive, how can i import it in spark-shell?

 On 19 Aug 2015, at 18:59, Yin Huai yh...@databricks.com wrote:
 
 Can you try to use HiveContext instead of SQLContext? Your query is trying to 
 create a table and persist the metadata of the table in metastore, which is 
 only supported by HiveContext.
 
 On Wed, Aug 19, 2015 at 8:44 AM, Yusuf Can Gürkan yu...@useinsider.com 
 mailto:yu...@useinsider.com wrote:
 Hello,
 
 I’m trying to create a table with sqlContext.sql method as below:
 
 val sc = new SparkContext()
 val sqlContext = new SQLContext(sc)
 
 import sqlContext.implicits._
 
 sqlContext.sql(s
 create table if not exists landing (
 date string,
 referrer string
 )
 partitioned by (partnerid string,dt string)
 row format delimited fields terminated by '\t' lines terminated by '\n'
 STORED AS TEXTFILE LOCATION 's3n://...' 
   ”)
 
 
 It gives error on spark-submit:
 
 Exception in thread main java.lang.RuntimeException: [2.1] failure: 
 ``with'' expected but identifier create found
 
 create external table if not exists landing (
 
 ^
   at scala.sys.package$.error(package.scala:27)
   at 
 org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
   at 
 org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
 
 
 
 What can be the reason??