mbutrovich commented on code in PR #5110: URL: https://github.com/apache/datafusion-comet/pull/5110#discussion_r3669248438
########## docs/source/user-guide/latest/compatibility/operators.md: ########## @@ -19,6 +19,23 @@ under the License. # Operator Compatibility +## Sampling + +Comet runs `SampleExec` natively when sampling is performed without replacement, which covers +`DataFrame.sample`, SQL `TABLESAMPLE`, and `DataFrame.randomSplit`. The native implementation +reproduces Spark's per-row `XORShiftRandom` draw sequence, so for a given seed it selects the same +rows as Spark. + +Because the sampler consumes one random value per row within a partition, it selects the same rows +as Spark only when the rows reach it in the same order. Sampling directly above a scan, filter, or +projection meets that condition. Sampling above an operator where Comet may emit rows in a +different order than Spark, such as a join or an aggregate, still returns a valid sample of the +same expected size, but not necessarily the same rows Spark would have selected. + +Sampling with replacement (`df.sample(withReplacement = true, ...)`) falls back to Spark, because +it draws from a Poisson distribution that Comet does not implement natively Review Comment: For the record, I think I did implement this natively in my early drafts of range partitioning, when we did the sampling in native code. It's in that commit history somewhere if we ever want to resurrect it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
