avantgardnerio commented on PR #2216: URL: https://github.com/apache/datafusion-ballista/pull/2216#issuecomment-5258452567
@phillipleblanc please re-read the PR body, I've updated it with revised framing where it no longer tries to justify a "swap". TDigest is strictly faster in apples-to-apples scenarios. The only remaining argument (which I am not making) for a "swap" would be code-complexity vs the ~1% time I saw in h2o q8 that was spent sketching at all. I'm fine with 2 modes for `RuntimeStatsExec`, but I would suggest perhaps it just picks the best one automatically? I'm not sure what it would do if you chose TDigest on a nullable or non-f64 column? > Compare equal accuracy levels Claude did some math with the first iteration of this PR to guess at what that was (100/800), but this time around we ran some empirical tests. It turns out those numbers are pretty close. Perhaps I misunderstood you, but I think the test is fair. > Give both algorithms equally sorted inpu Done, please see the new table in the PR body. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
