Hi All,
I wonder if there is a way to query data frame views directly without going
through scheduler? for example.
say I have the following code
DataSet<Row> kafkaDf = session.readStream().format("kafka").load();
kafkaDf.createOrReplaceView("table")
Now Can I query the view "table" without going through job scheduling
process?
Flink, for example, has some called QueryableState and also a Client which
will ask the JobManager where QueryableState is and just get the value but
it is limited to just K/V queries. It would be a lot more powerful if that
state can be queried using SQL since that would eliminate writing to an
external data store at least for a streaming dataset.
The problem currently is I know that I can query the view "table" in spark
my submitting another Job but the response is not in milliseconds (other
words, not realtime) so we would have to write to external datastore
and query it from there.
Thanks!
