Re: SAS_TO_SPARK_SQL_(Could be a Bug?)

Hi Ajay Looking at spark code , i can see you used hive context. Can you try using sql context instead of hive context there? Thanks Deepak On Mon, Jun 13, 2016 at 10:15 PM, Ajay Chander wrote: > Hi Mohit, > > Thanks for your time. Please find my response below. > > Did

SAS_TO_SPARK_SQL_(Could be a Bug?)

Hi Mohit, Thanks for your time. Please find my response below. Did you try the same with another database? I do load the data from MySQL and SQL Server the same way(through SPARK SQL JDBC) which works perfectly alright. As a workaround you can write the select statement yourself instead of just

Re: SAS_TO_SPARK_SQL_(Could be a Bug?)

Looks like a bug in the code generating the SQL query…why would it be specific to SAS, I can’t guess. Did you try the same with another database? As a workaround you can write the select statement yourself instead of just providing the table name. > On Jun 11, 2016, at 6:27 PM, Ajay Chander

Re: SAS_TO_SPARK_SQL_(Could be a Bug?)

I tried implementing the same functionality through Scala as well. But no luck so far. Just wondering if anyone here tried using Spark SQL to read SAS dataset? Thank you Regards, Ajay On Friday, June 10, 2016, Ajay Chander wrote: > Mich, I completely agree with you. I

Re: SAS_TO_SPARK_SQL_(Could be a Bug?)

Mich, I completely agree with you. I built another Spark SQL application which reads data from MySQL and SQL server and writes the data into Hive(parquet+snappy format). I have this problem only when I read directly from remote SAS system. The interesting part is I am using same driver to read

Re: SAS_TO_SPARK_SQL_(Could be a Bug?)

I personally use Scala to do something similar. For example here I extract data from an Oracle table and store in ORC table in Hive. This is compiled via sbt as run with SparkSubmit. It is similar to your code but in Scala. Note that I do not enclose my column names in double quotes. import

Re: SAS_TO_SPARK_SQL_(Could be a Bug?)

Hi Mich, Thanks for the response. If you look at my programs, I am not writings my queries to include column names in a pair of "". My driver in spark program is generating such query with column names in "" which I do not want. On the other hand, I am using the same driver in my pure Java

Re: SAS_TO_SPARK_SQL_(Could be a Bug?)

Assuming I understood your query, in Spark SQL (that is you log in to spark sql like spark-sql --master spark://:7077 you do not need double quotes around column names for sql to work spark-sql> select "hello from Mich" from oraclehadoop.sales limit 1; hello from Mich Anything between a pair of

Re: SAS_TO_SPARK_SQL_(Could be a Bug?)

Hi again, anyone in this group tried to access SAS dataset through Spark SQL ? Thank you Regards, Ajay On Friday, June 10, 2016, Ajay Chander wrote: > Hi Spark Users, > > I hope everyone here are doing great. > > I am trying to read data from SAS through Spark SQL and

SAS_TO_SPARK_SQL_(Could be a Bug?)

Hi Spark Users, I hope everyone here are doing great. I am trying to read data from SAS through Spark SQL and write into HDFS. Initially, I started with pure java program please find the program and logs in the attached file sas_pure_java.txt . My program ran successfully and it returned the