[
https://issues.apache.org/jira/browse/MRQL-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14987664#comment-14987664
]
ASF GitHub Bot commented on MRQL-79:
------------------------------------
GitHub user fegaras opened a pull request:
https://github.com/apache/incubator-mrql/pull/11
[MRQL-79] Add support for incremental query processing
The framework for incremental stream processing is described at
[streams15.pdf](http://lambda.uta.edu/streams15.pdf). Most of the changes are
at core/src/main/java/org/apache/mrql/Streaming.gen. The queries
queries/incremental-*.mrql describe various examples. For example, to run
k-mean clustering in incremental mode, first create the data:
`bin/mrql.spark -local queries/points.mrql 1000`
Then, process the data incrementally:
`bin/mrql.spark -local -stream 1000 queries/incremental-kmeans.mrql`
In a separate terminal, use `touch tmp/points.bin/part-00000` to change the
timestamp of the file to process the file again.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fegaras/incubator-mrql MRQL-79
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-mrql/pull/11.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11
----
commit 1cf73e2c8aa332a536548221aacc878e93174189
Author: fegaras <[email protected]>
Date: 2015-11-03T17:03:35Z
[MRQL-79] Add support for incremental query processing
----
> Add support for incremental query processing
> --------------------------------------------
>
> Key: MRQL-79
> URL: https://issues.apache.org/jira/browse/MRQL-79
> Project: MRQL
> Issue Type: New Feature
> Components: Run-Time/Spark, Streaming
> Affects Versions: 0.9.6
> Reporter: Leonidas Fegaras
> Assignee: Leonidas Fegaras
>
> This is a new feature for MRQL streaming: its task is to convert any
> stream-based MRQL query to an incremental query that merges the previous
> query results with the results of applying the query to the new data batches
> only. For example, it will be able to convert the MRQL PageRank query to an
> incremental PageRank query automatically. The basic idea was presented at
> ApacheCon'15 (page 28 in http://lambda.uta.edu/mrql-apachecon15.pdf ) as a
> future plan for MRQL. It will work on Spark Streaming mode for now, but later
> it will support Flink Streaming too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
