[
https://issues.apache.org/jira/browse/SPARK-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130880#comment-14130880
]
RJ Nowling commented on SPARK-3250:
-----------------------------------
Great work!
If these performance improvements hold up when implemented in Spark, this could
offer minibatch methods a fighting chance. In particular, we would need the
count the elements once and then we'd have faster sampling for the subsequent
iterations, especially if the underlying data structures can we coerced into
Arrays when we do the copy.
> More Efficient Sampling
> -----------------------
>
> Key: SPARK-3250
> URL: https://issues.apache.org/jira/browse/SPARK-3250
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: RJ Nowling
>
> Sampling, as currently implemented in Spark, is an O\(n\) operation. A
> number of stochastic algorithms achieve speed ups by exploiting O\(k\)
> sampling, where k is the number of data points to sample. Examples of such
> algorithms include KMeans MiniBatch (SPARK-2308) and Stochastic Gradient
> Descent with mini batching.
> More efficient sampling may be achievable by packing partitions with an
> ArrayBuffer or other data structure supporting random access. Since many of
> these stochastic algorithms perform repeated rounds of sampling, it may be
> feasible to perform a transformation to change the backing data structure
> followed by multiple rounds of sampling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
