Re: Pointing SparkSQL to existing Hive Metadata with data file locations in HDFS

t;SQL File" mode - /opt/cloudera/parcels/CDH/lib/spark/bin/spark-sql -f   get_names.hql From: Andrew Otto To: Sanjay Subramanian Cc: user Sent: Thursday, May 28, 2015 7:26 AM Subject: Re: Pointing SparkSQL to existing Hive Metadata with data file locations in HDFS

Re: Pointing SparkSQL to existing Hive Metadata with data file locations in HDFS

> val sqlContext = new HiveContext(sc) > val schemaRdd = sqlContext.sql("some complex SQL") It mostly works, but have been having issues with tables that contains a large amount of data: https://issues.apache.org/jira/browse/SPARK-6910 > On

Re: Pointing SparkSQL to existing Hive Metadata with data file locations in HDFS

Yes, you are at right path. Only thing to remember is placing hive site XML to correct path so spark can talk to hive metastore. Best Ayan On 28 May 2015 10:53, "Sanjay Subramanian" wrote: > hey guys > > On the Hive/Hadoop ecosystem we have using Cloudera distribution CDH 5.2.x > , there are abo

RE: Pointing SparkSQL to existing Hive Metadata with data file locations in HDFS

Yes, but be sure you put the hive-site.xml under your class path. Any problem you meet? Cheng Hao From: Sanjay Subramanian [mailto:sanjaysubraman...@yahoo.com.INVALID] Sent: Thursday, May 28, 2015 8:53 AM To: user Subject: Pointing SparkSQL to existing Hive Metadata with data file locations in