GitHub user ZiyueHuang opened a pull request:
https://github.com/apache/spark/pull/18349
[SPARK-20927][SS] Change some operators in Dataset to no-op for a streaming
query.
## What changes were proposed in this pull request?
Change some operators(persist, unpersist, checkpoint) in Dataset to no-op
(do nothing but log a warning) for a streaming query.
## How was this patch tested?
```scala
df = spark.readStream.json(...)
val dfCounts = df.persist().unpersist().checkpoint().groupBy().count()
val query =
dfCounts.writeStream.outputMode("complete").format("console").start()
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ZiyueHuang/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18349.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18349
----
commit 8358f576bd255db458c630fc38537dd93c695246
Author: Ziyue Huang <zyhuan...@gmail.com>
Date: 2017-06-15T06:05:18Z
[SPARK-20927][SS] Change some operators to no-op for a streaming query
without throwing an exception.
commit 1713c568de223a78bc6db4455e8ea38c4f2d2267
Author: ZiyueHuang <zyhuan...@gmail.com>
Date: 2017-06-15T06:32:57Z
Revert "[SPARK-20927][SS] Change some operators to no-op for a streaming
query without throwing an exception."
This reverts commit 8358f576bd255db458c630fc38537dd93c695246.
revert
commit 372ada118883b7ac8924ff20bd76e4bd15b47d6f
Author: ZiyueHuang <zyhuan...@gmail.com>
Date: 2017-06-15T06:39:28Z
[SPARK-20927][SS] Change some operators to no-op in streaming queries.
commit f68e6b738d9330409850556c40d626bfb17a561e
Author: ZiyueHuang <zyhuan...@gmail.com>
Date: 2017-06-19T03:10:43Z
scalastyle fix
commit f127ada3281442fc1def31328deaab7463f88531
Author: ZiyueHuang <zyhuan...@gmail.com>
Date: 2017-06-19T03:50:34Z
comment fix
commit bb1fc6f23d9a74f649aef7b6707a82f45df3eff3
Author: Ziyue Huang <zyhuan...@gmail.com>
Date: 2017-06-19T04:17:59Z
Merge pull request #1 from ZiyueHuang/dev
[SPARK-20927][SS] Change some operators to no-op in streaming queries.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
