Thank you Todd, How is the sparkstreaming-sql project different from starting a thrift server on a streaming app ?
Thanks again. Daniel On Thu, Aug 6, 2015 at 1:53 AM, Todd Nist <tsind...@gmail.com> wrote: > Hi Danniel, > > It is possible to create an instance of the SparkSQL Thrift server, > however seems like this project is what you may be looking for: > > https://github.com/Intel-bigdata/spark-streamingsql > > Not 100% sure of your use case is, but you can always convert the data > into DF then issue a query against it. If you want other systems to be > able to query it then there are numerous connectors to store data into > Hive, Cassandra, HBase, ElasticSearch, .... > > To create a instance of a thrift server with its own SQL Context you would > do something like the following: > > import org.apache.spark.{SparkConf, SparkContext} > > import org.apache.spark.sql.hive.HiveContext > import org.apache.spark.sql.hive.HiveMetastoreTypes._ > import org.apache.spark.sql.types._ > import org.apache.spark.sql.hive.thriftserver._ > > > object MyThriftServer { > > val sparkConf = new SparkConf() > // master is passed to spark-submit, but could also be specified > explicitely > // .setMaster(sparkMaster) > .setAppName("My ThriftServer") > .set("spark.cores.max", "2") > val sc = new SparkContext(sparkConf) > val sparkContext = sc > import sparkContext._ > val sqlContext = new HiveContext(sparkContext) > import sqlContext._ > import sqlContext.implicits._ > > makeRDD((1,"hello") :: (2,"world") > ::Nil).toDF.cache().registerTempTable("t") > > HiveThriftServer2.startWithContext(sqlContext) > } > > Again, I'm not really clear what your use case is, but it does sound like > the first link above is what you may want. > > -Todd > > On Wed, Aug 5, 2015 at 1:57 PM, Daniel Haviv < > daniel.ha...@veracity-group.com> wrote: > >> Hi, >> Is it possible to start the Spark SQL thrift server from with a streaming >> app so the streamed data could be queried as it's goes in ? >> >> Thank you. >> Daniel >> > >
