Hey, This is my spark-env:
# Add Hadoop libraries to Spark classpath SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HADOOP_HOME}/*:${HADOOP_HOME}/../hadoop-hdfs/*:${HADOOP_HOME}/../hadoop-mapreduce/*:${HADOOP_HOME}/../hadoop-yarn/*" LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HADOOP_HOME}/lib/native" # Add Hive conf to Spark classpath SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HIVE_CONF_DIR}" # Add Hadoop LZO library to Spark classpath and native library to library path SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HADOOP_HOME}/../hadoop-lzo/lib/*" LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HADOOP_HOME}/../hadoop-lzo/lib/native" # Add EMRFS libraries to Spark classpath SPARK_CLASSPATH="${SPARK_CLASSPATH}:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*" export SPARK_CLASSPATH export LD_LIBRARY_PATH I think hive is on class path, should i do another thing? > On 19 Aug 2015, at 20:12, Eric Walker <e...@node.io> wrote: > > Hi, > > On EMR, take a look at the "SPARK_CLASSPATH" variable in > /home/hadoop/spark/conf/spark-env.sh. You may need to add a jar file to it. > > Regards, > Eric > > > On Wed, Aug 19, 2015 at 11:03 AM, Yusuf Can Gürkan <yu...@useinsider.com > <mailto:yu...@useinsider.com>> wrote: > Hey Yin, > > Thanks for answer. I thought that this could be problem but i can not create > HiveContext because i can not import org.apache.spark.sql.hive.HiveContext. > It does not see this package. > > I read that i should build spark with -PHive but i’m running on Amazon EMR > 1.4.1 and on spark-shell i can import hive package but can not do the same on > spark-submit. Do you have any idea why? Because if it’s related to build with > -PHive, how can i import it in spark-shell? > >> On 19 Aug 2015, at 18:59, Yin Huai <yh...@databricks.com >> <mailto:yh...@databricks.com>> wrote: >> >> Can you try to use HiveContext instead of SQLContext? Your query is trying >> to create a table and persist the metadata of the table in metastore, which >> is only supported by HiveContext. >> >> On Wed, Aug 19, 2015 at 8:44 AM, Yusuf Can Gürkan <yu...@useinsider.com >> <mailto:yu...@useinsider.com>> wrote: >> Hello, >> >> I’m trying to create a table with sqlContext.sql method as below: >> >> val sc = new SparkContext() >> val sqlContext = new SQLContext(sc) >> >> import sqlContext.implicits._ >> >> sqlContext.sql(s""" >> create table if not exists landing ( >> date string, >> referrer string >> ) >> partitioned by (partnerid string,dt string) >> row format delimited fields terminated by '\t' lines terminated by '\n' >> STORED AS TEXTFILE LOCATION 's3n://...' <> >> "”") >> >> >> It gives error on spark-submit: >> >> Exception in thread "main" java.lang.RuntimeException: [2.1] failure: >> ``with'' expected but identifier create found >> >> create external table if not exists landing ( >> >> ^ >> at scala.sys.package$.error(package.scala:27) >> at >> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36) >> at >> org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67) >> >> >> >> What can be the reason?? >> > >