At first glance, it looks like the only streaming data sources available out of
the box from the github master branch are
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
and
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala
. Out of the Jira epic for Structured Streaming
https://issues.apache.org/jira/browse/SPARK-8360 it would seem the still-open
https://issues.apache.org/jira/browse/SPARK-10815 "API design: data sources and
sinks" is relevant here.
In short, it would seem the code is not there yet to create a Kafka-fed
Dataframe/Dataset that can be queried with Structured Streaming; or if it is,
it's not obvious how to write such code.
From: Anthony May <anthony...@gmail.com>
To: Deepak Sharma <deepakmc...@gmail.com>; Sunita Arvind
<sunitarv...@gmail.com>
Cc: "user@spark.apache.org" <user@spark.apache.org>
Sent: Friday, May 6, 2016 11:50 AM
Subject: Re: Adhoc queries on Spark 2.0 with Structured Streaming
Yeah, there isn't even a RC yet and no documentation but you can work off the
code base and test suites:
https://github.com/apache/spark
And this might help:
https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/streaming/DataFrameReaderWriterSuite.scala
On Fri, 6 May 2016 at 11:07 Deepak Sharma <deepakmc...@gmail.com> wrote:
Spark 2.0 is yet to come out for public release.
I am waiting to get hands on it as well.
Please do let me know if i can download source and build spark2.0 from github.
Thanks
Deepak
On Fri, May 6, 2016 at 9:51 PM, Sunita Arvind <sunitarv...@gmail.com> wrote:
Hi All,
We are evaluating a few real time streaming query engines and spark is my
personal choice. The addition of adhoc queries is what is getting me further
excited about it, however the talks I have heard so far only mention about it
but do not provide details. I need to build a prototype to ensure it works for
our use cases.
Can someone point me to relevant material for this.
regards
Sunita
--
Thanks
Deepak
www.bigdatabig.com
www.keosha.net
