GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/17488
[SPARK-20165][SS] Resolve state encoder's deserializer in driver in
FlatMapGroupsWithStateExec
## What changes were proposed in this pull request?
- Encoder's deserializer must be resolved at the driver where the class is
defined. Otherwise there are corner cases using nested classes where resolving
at the executor can fail.
- Fix flaky test related to processing time timeout. The flakiness is
caused because the test thread (that adds data to memory source) has a race
condition with the streaming query thread. When testing the manual clock, the
goal is to add data and increment clock together atomically, such that a
trigger sees new data AND updated clock simultaneously (both or none). This fix
adds additional synchronization in when adding data; it makes sure that the
streaming query thread is waiting on the manual clock to be incremented (so no
batch is currently running) before adding data.
## How was this patch tested?
Multiple runs on existing unit tests
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-20165
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17488.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17488
----
commit 468c037ca927b43e52015a378b8c70145a99c5da
Author: Tathagata Das <tathagata.das1...@gmail.com>
Date: 2017-03-30T23:34:12Z
Fixed bugs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
