tdcmeehan commented on PR #58946:
URL: https://github.com/apache/spark/pull/58946#issuecomment-5770544748
Thanks for picking this up, and for the pointer to the SPARK-56822 umbrella.
I have reviewed the change; my comments are in the review. The main one is that
the new `toDouble`/`fromDouble` encoding in `EstimationUtils` does not keep the
property its comment states -- that distinct nanosecond values stay
distinguishable and come back unchanged through `fromDouble` -- once
`epochMicros` exceeds 2^43 (1970-04-12), which covers every realistic
timestamp. Relative order is unaffected, since `toDouble` stays monotone, so
what is lost is resolution rather than sort order. The new
`EstimationUtilsSuite` cases pass because they sit at `epochMicros` 100 and
-100, inside the window where the fraction survives.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:5763723241","thread_id":"conversation:5763723241","verdict_sha256":"373510bc4f00787cf412c0d7a490787c05aed02da574143e10a9d91d40d10ac2"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
