GitHub user mukulmurthy opened a pull request: https://github.com/apache/spark/pull/21662
[SPARK-24662][SQL][SS] Support limit in structured streaming ## What changes were proposed in this pull request? Support the LIMIT operator in structured streaming. For streams in append or complete output mode, a stream with a LIMIT operator will return no more than the specified number of rows. LIMIT is still unsupported for the update output mode. ## How was this patch tested? New and existing unit tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mukulmurthy/oss-spark SPARK-24662 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21662.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21662 ---- commit 94b93f735b32fc6be0add4ac710987f626e46e1a Author: Mukul Murthy <mukul.murthy@...> Date: 2018-06-26T23:06:45Z Revert "[SPARK-24525][SS] Provide an option to limit number of rows in a MemorySink" This reverts commit e4fee395ecd93ad4579d9afbf0861f82a303e563. commit 1ba894a83297ee1e728511c4e90e10e1abc8fcaf Author: Mukul Murthy <mukul.murthy@...> Date: 2018-06-26T23:38:32Z Add tests for streaming with LIMIT commit b006f4a739eac54f191a904b64ad814881fa70e5 Author: Mukul Murthy <mukul.murthy@...> Date: 2018-06-27T17:11:32Z Updated the unsupported operation checker and added test for Update mode commit 66e1073b9e87f0fd604c8d43651264753a701279 Author: Mukul Murthy <mukul.murthy@...> Date: 2018-06-27T23:33:05Z Scaffold StreamingLimitExec and insert it into query plan for streams commit 585e5b3cf7fb6f182c02c7204ca80e6da78b203c Author: Mukul Murthy <mukul.murthy@...> Date: 2018-06-28T18:00:06Z Make StreamingLimitExec read and store state commit 904d4379424e351711034d7294baa1a4955625cd Author: Mukul Murthy <mukul.murthy@...> Date: 2018-06-28T19:51:42Z Implement the limit and add more tests commit 82b72286cc6eefdee078d99668dfb2b44f634ed5 Author: Mukul Murthy <mukul.murthy@...> Date: 2018-06-28T22:18:18Z Cleaned up implementation, better testing ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org