Hi Danniel, It is possible to create an instance of the SparkSQL Thrift server, however seems like this project is what you may be looking for:
https://github.com/Intel-bigdata/spark-streamingsql Not 100% sure of your use case is, but you can always convert the data into DF then issue a query against it. If you want other systems to be able to query it then there are numerous connectors to store data into Hive, Cassandra, HBase, ElasticSearch, .... To create a instance of a thrift server with its own SQL Context you would do something like the following: import org.apache.spark.{SparkConf, SparkContext} import org.apache.spark.sql.hive.HiveContext import org.apache.spark.sql.hive.HiveMetastoreTypes._ import org.apache.spark.sql.types._ import org.apache.spark.sql.hive.thriftserver._ object MyThriftServer { val sparkConf = new SparkConf() // master is passed to spark-submit, but could also be specified explicitely // .setMaster(sparkMaster) .setAppName("My ThriftServer") .set("spark.cores.max", "2") val sc = new SparkContext(sparkConf) val sparkContext = sc import sparkContext._ val sqlContext = new HiveContext(sparkContext) import sqlContext._ import sqlContext.implicits._ makeRDD((1,"hello") :: (2,"world") ::Nil).toDF.cache().registerTempTable("t") HiveThriftServer2.startWithContext(sqlContext) } Again, I'm not really clear what your use case is, but it does sound like the first link above is what you may want. -Todd On Wed, Aug 5, 2015 at 1:57 PM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi, > Is it possible to start the Spark SQL thrift server from with a streaming > app so the streamed data could be queried as it's goes in ? > > Thank you. > Daniel >
