GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/18717
[SPARK-21510] [SQL] Add isMaterialized() and eager persist() to Dataset APIs
### What changes were proposed in this pull request?
Currently, when using Spark, the beginners do not realize our persist API
is lazy. They do not know what is the most efficient way to materialize it.
Sometimes, they just use collect(), which is very expensive when the data set
is big. This PR is to add a simple API to enable users to do eager persist.
In addition, we also need another Dataset API to verify whether the Dataset
has been cached and materialized.
### How was this patch tested?
Added two test cases and simplify the existing test cases by using the
newly added API.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark eagerPersist
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18717.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18717
----
commit 7f82e38f8edca611cdf55266ff07ac4523799603
Author: gatorsmile <gatorsm...@gmail.com>
Date: 2017-07-23T00:17:26Z
fix
commit 80e7123c3905c44776233e0a56a8d8315f1313a0
Author: gatorsmile <gatorsm...@gmail.com>
Date: 2017-07-23T04:17:56Z
fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
