Spenserrrr commented on PR #58306: URL: https://github.com/apache/spark/pull/58306#issuecomment-5429642382
Hi @zhengruifeng! This is a PR fixing the`floor_divide`. `1.0 // 0.1` was returning 10, but pandas and NumPy give 9. This is because the division rounds the quotient up before the floor runs. Right now, the code calculates the quotient out of the remainder the way NumPy does. And integral operands were cast to double first, which loses the low bits above 2^53, so they now divide in integer space. Could you take a look when you have time? One thing is worth your attention. The mapping I fixed here is the unreachable one, since `np.floor_divide` goes through the `floordiv` dunder first. The same two bugs are live in `data_type_ops/num_ops.py`, where `ps.Series([1.0]) // 0.1` gives 10.0 against pandas' 9.0 and `psser // 2` is off by one above 2^53. I will fix that as a separate PR with its own JIRA, since it changes what users see on a path that has behaved this way for a long time. Also please tell me if you would rather leave it alone. Thanks! cc @Yicong-Huang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
