deshanxiao created SPARK-31140:
----------------------------------
Summary: Support Quick sample in RDD
Key: SPARK-31140
URL: https://issues.apache.org/jira/browse/SPARK-31140
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.0.0
Reporter: deshanxiao
RDD.sample use *filter* to read the raw data. It means that if the raw data is
very huge, we must cost too much time to read it. We can filter the raw
partition to speed up the processing of sample.
{code:java}
override def compute(splitIn: Partition, context: TaskContext): Iterator[U] =
{
val split = splitIn.asInstanceOf[PartitionwiseSampledRDDPartition]
val thisSampler = sampler.clone
thisSampler.setSeed(split.seed)
thisSampler.sample(firstParent[T].iterator(split.prev, context))
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
