GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/11645
[WIP][SPARK-13809][SQL] State store for streaming aggregations ## What changes were proposed in this pull request? In this PR, I am implementing a new abstraction for management of streaming state data - State Store. It is a key-value store for persisting running aggregates for aggregate operations in streaming dataframes. The motivation and design is discussed here. https://docs.google.com/document/d/1-ncawFx8JS5Zyfq1HAEGBx56RDet9wfVp_hDM8ZL254/edit# ## How was this patch tested? Unit tests (WIP) ## TODO [ ] Unit tests - [ ] StateStoreCoordinator - [ ] StateStoreRDD fault-tolerance You can merge this pull request into a Git repository by running: $ git pull https://github.com/tdas/spark state-store Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11645.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11645 ---- commit c0194744378c2b1827aa6bb3a6109b3549710a44 Author: Tathagata Das <tathagata.das1...@gmail.com> Date: 2016-03-10T23:02:37Z First draft of StateStore commit 4f8dade98ab2720ffab78a6ace567b9705f1b488 Author: Tathagata Das <tathagata.das1...@gmail.com> Date: 2016-03-11T01:50:50Z Updated tests commit f417bde8bd65572e78125c7aa900221fa0bc4d7c Author: Tathagata Das <tathagata.das1...@gmail.com> Date: 2016-03-11T04:24:39Z Added basic unit test for StateStoreRDD ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org