GitHub user xuanyuanking opened a pull request: https://github.com/apache/spark/pull/22583
[SPARK-10816][SS] SessionWindow support for Structure Streaming ## What changes were proposed in this pull request? As the discussion in [SPARK-10816](https://issues.apache.org/jira/browse/SPARK-10816), here we add the support for session window support in Structure Streaming. For convenient review, this PR can be split to several part: - Implement for session_window build-in function in dataset api, add related analysis rule and exec node. - Implement for integrating with streaming frame work, including the SessionWindowStateStoreRestoreExec/SessionWindowMergeExec/SessionWindowStateStoreSaveExec. ## How was this patch tested? Test for dataset\sql only and end-to-end streaming tests. Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuanyuanking/spark SPARK-10816 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22583.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22583 ---- commit 5e4034ff26f3bab65d44a142aa29ec5b2082e81f Author: liyuanjian <liyuanjian@...> Date: 2018-08-02T09:38:04Z INF-SPARK-1474: add build.sh and BCLOUD for new branch commit 2d943d5a4e7659a22686ac5490d8aecd63d939ba Author: liyuanjian <liyuanjian@...> Date: 2018-08-02T10:16:22Z acu-bsc-15: add session window expression commit 55b8196e6622710dfed3b25305a3ec408af772fb Author: liyuanjian <liyuanjian@...> Date: 2018-08-16T09:01:33Z acu-bsc-19: Add support in logical\pysical plan for session window, add UT in DataFrameSessionWindowingSuite commit 05dcd7f3e5572ff429632407099afa9d536e36e3 Author: Yuanjian Li <xyliyuanjian@...> Date: 2018-09-28T06:47:56Z Fix for spark master baned transform in Analyzer commit b138f09479523d6386137c4c6a0b7dbe03b97273 Author: Yuanjian Li <xyliyuanjian@...> Date: 2018-08-22T13:18:43Z Addression comments and code clean commit a2517b610cbb38123d1edf4eda2a5186d44b1a22 Author: liyuanjian <liyuanjian@...> Date: 2018-08-23T06:44:15Z Address comment and bug fix commit 29567883fb76911b0bc3614cf5b0c910d6a0b200 Author: huangtengfei02 <huangtengfei02@...> Date: 2018-08-29T09:11:22Z acu-bsc-40: add SessionWindowMergeExec Change-Id: Ie9669ee7827f0e52d2837ddd8a840a07543850ba commit cb770c76147bbc20bb3627c913c3526b2f8dc161 Author: zhuliangchang <zhuliangchang@...> Date: 2018-09-05T04:10:33Z acu-bsc-20, StateStore management of SessionWindow 1. acu-bsc-21, SessionWindowStateStoreRestore by huangtengfei02 <huangtengfe...@baidu.com> 2. acu-bsc-22, SessionWindowStateStoreSave of complete / append mode 3. acu-bsc-18, new Strategy for Streaming query with SessionWindow 4. acu-bsc-17, generate new aggregate physical plan in AggUtils commit d65015e970a126e6f128aab63a47f436f272028c Author: Yuanjian Li <xyliyuanjian@...> Date: 2018-09-28T09:42:51Z Remove internal build script ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org