GitHub user mukulmurthy opened a pull request:
https://github.com/apache/spark/pull/21662
[SPARK-24662][SQL][SS] Support limit in structured streaming
## What changes were proposed in this pull request?
Support the LIMIT operator in structured streaming.
For streams in append or complete output mode, a stream with a LIMIT
operator will return no more than the specified number of rows. LIMIT is still
unsupported for the update output mode.
## How was this patch tested?
New and existing unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mukulmurthy/oss-spark SPARK-24662
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21662.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21662
----
commit 94b93f735b32fc6be0add4ac710987f626e46e1a
Author: Mukul Murthy <mukul.murthy@...>
Date: 2018-06-26T23:06:45Z
Revert "[SPARK-24525][SS] Provide an option to limit number of rows in a
MemorySink"
This reverts commit e4fee395ecd93ad4579d9afbf0861f82a303e563.
commit 1ba894a83297ee1e728511c4e90e10e1abc8fcaf
Author: Mukul Murthy <mukul.murthy@...>
Date: 2018-06-26T23:38:32Z
Add tests for streaming with LIMIT
commit b006f4a739eac54f191a904b64ad814881fa70e5
Author: Mukul Murthy <mukul.murthy@...>
Date: 2018-06-27T17:11:32Z
Updated the unsupported operation checker and added test for Update mode
commit 66e1073b9e87f0fd604c8d43651264753a701279
Author: Mukul Murthy <mukul.murthy@...>
Date: 2018-06-27T23:33:05Z
Scaffold StreamingLimitExec and insert it into query plan for streams
commit 585e5b3cf7fb6f182c02c7204ca80e6da78b203c
Author: Mukul Murthy <mukul.murthy@...>
Date: 2018-06-28T18:00:06Z
Make StreamingLimitExec read and store state
commit 904d4379424e351711034d7294baa1a4955625cd
Author: Mukul Murthy <mukul.murthy@...>
Date: 2018-06-28T19:51:42Z
Implement the limit and add more tests
commit 82b72286cc6eefdee078d99668dfb2b44f634ed5
Author: Mukul Murthy <mukul.murthy@...>
Date: 2018-06-28T22:18:18Z
Cleaned up implementation, better testing
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
