GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/18387
Validate sampling fraction in logical operator level
## What changes were proposed in this pull request?
Currently the validation of sampling fraction in dataset is incomplete.
As an improvement, validate sampling ratio in logical operator level:
1) if with replacement: ratio should be nonnegative
2) else: ratio should be on interval [0, 1]
Also add test cases for the validation.
## How was this patch tested?
integration tests
@gatorsmile @cloud-fan
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gengliangwang/spark sample_ratio_validate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18387.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18387
----
commit e4dc81b998b6a2fe8f8ec494cb6e34d64530dbb7
Author: Wang Gengliang <[email protected]>
Date: 2017-06-21T23:29:53Z
Validate sampling ratio in logical operator level
1) if with replacement: ratio should be nonneative
2) else: ratio should be on interval [0, 1]
commit fe7f087aaf5ef180b8cd1d3bce43bbe28e92515d
Author: Wang Gengliang <[email protected]>
Date: 2017-06-22T00:31:58Z
add test cases
commit 5417944b1a01e299961336c1135c3fd10934f7c7
Author: Wang Gengliang <[email protected]>
Date: 2017-06-22T06:43:41Z
revise test cases
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
