[
https://issues.apache.org/jira/browse/MRQL-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307588#comment-14307588
]
Robert Metzger commented on MRQL-66:
------------------------------------
Cool!
I've forwarded the JIRA to the Flink dev list. Maybe somebody from Flink is
interested in helping or giving advice.
> Add support for MRQL streaming in Flink streaming mode
> ------------------------------------------------------
>
> Key: MRQL-66
> URL: https://issues.apache.org/jira/browse/MRQL-66
> Project: MRQL
> Issue Type: New Feature
> Components: Run-Time/Flink, Streaming
> Affects Versions: 0.9.6
> Reporter: Leonidas Fegaras
> Priority: Critical
>
> The new extension, MRQL Streaming, works fine with Spark Streaming (see
> MRQL-63) but it would be nice if we make it work with Flink Streaming too. It
> was easy to make it work with Spark Streaming: Data in one sliding window in
> a Spark's DStream is viewed as an RDD. So a DStream can be viewed as a
> continuous sequence of RDDs. A DStream has a method foreachRDD that applies a
> function to each RDD in the stream. So to implement MRQL Streaming, we just
> had to use the MRQL Spark evaluator (a function from RDD to RDD) as an
> argument to foreachRDD. For Flink Streaming, the implementation will be more
> complicated. A Flink Streaming DataStream doesn't provide a hook to a DataSet
> object. I am guessing that this is because Flink Streaming is far more
> general than Spark Streaming (it's not just sliding windows) and because
> Flink Streaming needs to do special optimizations. So we need to copy the
> FlinkEvaluator class into a new class FlinkStreaming and change all methods
> to be on DataStream instead of DataSet. Many DataSet methods have an
> equivalent in DataStream but some are missing. I have already provided the
> input formats for streaming (method FlinkStreaming.stream_source) but we need
> to write a stream evaluator for MRQL plans.
> Any volunteer?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
