Re: query avro hive table in spark sql

2015-08-28 Thread Giri P
Any idea what causing this error 15/08/28 21:03:03 WARN scheduler.TaskSetManager: Lost task 34.0 in stage 9.0 (TID 20, dtord01hdw0228p.dc.dotomi.net): java.lang.RuntimeException: cannot find field message_campaign_id from [0:error_error_error_error_error_error_error, 1:cannot_determine_schema,

RE: query avro hive table in spark sql

2015-08-27 Thread java8964
, 26 Aug 2015 17:48:44 -0700 Subject: Re: query avro hive table in spark sql To: gpatc...@gmail.com CC: user@spark.apache.org I'd suggest looking at http://spark-packages.org/package/databricks/spark-avro On Wed, Aug 26, 2015 at 11:32 AM, gpatcham gpatc...@gmail.com wrote: Hi, I'm trying to query

Re: query avro hive table in spark sql

2015-08-27 Thread Giri P
, can you post the whole stack for that error? Most of our datasets are also in AVRO format. Yong -- Date: Thu, 27 Aug 2015 09:45:45 -0700 Subject: Re: query avro hive table in spark sql From: gpatc...@gmail.com To: java8...@hotmail.com CC: mich

RE: query avro hive table in spark sql

2015-08-27 Thread java8964
library mismatch causing it, but not sure. I never faced your 2nd problem, can you post the whole stack for that error? Most of our datasets are also in AVRO format. Yong Date: Thu, 27 Aug 2015 09:45:45 -0700 Subject: Re: query avro hive table in spark sql From: gpatc...@gmail.com To: java8

Re: query avro hive table in spark sql

2015-08-27 Thread Giri P
for our experience, but still, some non-tech people just want to use as a SQL shell in spark, like HIVE-CLI. Yong -- From: mich...@databricks.com Date: Wed, 26 Aug 2015 17:48:44 -0700 Subject: Re: query avro hive table in spark sql To: gpatc...@gmail.com CC: user

Re: query avro hive table in spark sql

2015-08-27 Thread Giri P
schema using url. I'm stuck and couldn't find more info on this. Any pointers ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/query-avro-hive-table-in-spark-sql-tp24462.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: query avro hive table in spark sql

2015-08-27 Thread Michael Armbrust
BTY, spark-avro works great for our experience, but still, some non-tech people just want to use as a SQL shell in spark, like HIVE-CLI. To clarify: you can still use the spark-avro library with pure SQL. Just use the CREATE TABLE ... USING com.databricks.spark.avro OPTIONS (path '...')

Re: query avro hive table in spark sql

2015-08-27 Thread ponkin
Can you select something from this table using Hive? And also could you post your spark code which leads to this exception. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/query-avro-hive-table-in-spark-sql-tp24462p24468.html Sent from the Apache Spark User

query avro hive table in spark sql

2015-08-26 Thread gpatcham
) Its not able to determine schema. Hive table is pointing to avro schema using url. I'm stuck and couldn't find more info on this. Any pointers ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/query-avro-hive-table-in-spark-sql-tp24462.html Sent from

Re: query avro hive table in spark sql

2015-08-26 Thread Michael Armbrust
on this. Any pointers ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/query-avro-hive-table-in-spark-sql-tp24462.html Sent from the Apache Spark User List mailing list archive at Nabble.com