GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/19924
[SPARK-22187][SS][REVERT] Revert change in state row format for
mapGroupsWithState
## What changes were proposed in this pull request?
#19416 changed the format in which rows were encoded in the state store.
However, this can break existing streaming queries with the old format in
unpredictable ways (potentially crashing the JVM). Hence I am reverting this
for now. This will be re-applied in the future after we start saving more
metadata in checkpoints to signify which version of state row format the
existing streaming query is running. Then we can decode old and new formats
accordingly.
## How was this patch tested?
Existing tests.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-22187-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19924.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19924
commit 4a606de0c797fa6e8338d9a4987ff1a35e954c54
Author: Tathagata Das
Date: 2017-11-08T08:00:23Z
Revert row format
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org