Github user thvasilo commented on a diff in the pull request:
https://github.com/apache/flink/pull/949#discussion_r36969808
--- Diff: flink-java/src/main/java/org/apache/flink/api/java/DataSet.java
---
@@ -1057,7 +1061,68 @@ public long count() throws Exception {
public UnionOperator<T> union(DataSet<T> other){
return new UnionOperator<T>(this, other,
Utils.getCallLocationName());
}
+
+ //
--------------------------------------------------------------------------------------------
+ // Sample
+ //
--------------------------------------------------------------------------------------------
+
+ /**
+ * Generate a sample of DataSet by the probability fraction of each
element.
+ *
+ * @param withReplacement Whether element can be selected more than
once.
+ * @param fraction Probability that each element is chosen,
should be [0,1] without replacement,
+ * and [0, â) with replacement. While fraction
is larger than 1, the elements are
+ * expected to be selected multi times into
sample on average.
+ * @return The sampled DataSet
+ */
+ public MapPartitionOperator<T, T> sample(final boolean withReplacement,
final double fraction) {
+ return sample(withReplacement, fraction, Utils.RNG.nextLong());
+ }
+
+ /**
+ * Generate a sample of DataSet by the probability fraction of each
element.
+ *
+ * @param withReplacement Whether element can be selected more than
once.
+ * @param fraction Probability that each element is chosen,
should be [0,1] without replacement,
+ * and [0, â) with replacement. While fraction
is larger than 1, the elements are
+ * expected to be selected multi times into
sample on average.
+ * @param seed random number generator seed.
+ * @return The sampled DataSet
+ */
+ public MapPartitionOperator<T, T> sample(final boolean withReplacement,
final double fraction, final long seed) {
+ return mapPartition(new SampleWithFraction<T>(withReplacement,
fraction, seed));
+ }
+
+ /**
+ * Generate a sample of DataSet which contains fixed size elements.
+ *
+ * @param withReplacement Whether element can be selected more than
once.
+ * @param numSample The expected sample size.
+ * @return The sampled DataSet
+ */
--- End diff --
Maybe we want to include a note that this kind of sampling currently takes
2 passes over the data, and recommend using fraction unless exact precision is
necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
