Re: Custom Data Source for getting data from Rest based services

[smazumder]

@sathich

Here are my thoughts on your points -

1. Yes this should be able to handle any complex json structure returned by
the target rest API. Essentially what it would be returning is Rows of that
complex structure. Then one can use Spark SQL to further flatten it using
the functions like inline, explode, etc.

2. In my current implementation I have kept an option as "callStrictlyOnce".
This will ensure that the REST API is called only once for each set of
parameter values and the result would be persisted/cached for next time use.

3. I'm not sure what exactly you have in mind regarding extending this to
Spark Streaming. As such this cannot be used as a Spark Streaming receiver
right now as this does not implement the necessary interfaces for a custom
streaming receiver. But you can use this within your Spark Streaming
application as a regular Data Source to merge the data you are receiving
from streaming source.

Regards,
Sourav



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Spark SQL is not able to connect to hive metastore

[smazumder]

HI,

I'm trying to execute simple sql statement from spark-shell

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) - This one
executes properly.

Next I'm trying -
sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")

This keeps on trying to connect Metastore but could nener. The error I'm
getting is -

Trying to connect to metastore with URI thrift://.com:9083
set_ugi() not successful, Likely cause: new client talking to old server.
Continuing without it.
org.apache.thrift.transport.TTransportException: java.net.SocketException:
Connection reset

Any idea ?

I'm using Spark 1.2.1 and Hive is 0.14.

Regards,
Sourav




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-is-not-able-to-connect-to-hive-metastore-tp22916.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Using Spark SQL in Spark1.2.1 with Hive 0.14

[smazumder]

Hi,

I'm trying to execute queries from beeline to Hive 0.14 from Spark SQL
(1.2.1).

A simple query like 'show tables' or 'create schema ' doe not
return back at all.

Do I need to update to Spark 1.3 for this to work with 0.14 ? Is there any
other alternatives ?

Regards,
Sourav



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Using-Spark-SQL-in-Spark1-2-1-with-Hive-0-14-tp22912.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org