srowen commented on PR #44690: URL: https://github.com/apache/spark/pull/44690#issuecomment-1968192652
float -> integer conversion in the JVM always truncates so yes (for positive numbers) you are rounding down by doing this. I think my point is, the fix actually has nothing to do with working in integers; it is the 'rounding down' by 1 ulp to make sure that (for example) 1.0/11.0 is <= 1/11th. I think this would have worked as well if you shaved 1 ulp off the inputs, for instance. It may not be worth pursuing this any further though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
