tustvold commented on PR #4640: URL: https://github.com/apache/arrow-rs/pull/4640#issuecomment-1666514768
@westonpace I'm having a hard time following the logic behind adjustScaleIfNeeded. I can understand truncating the scale of the output in order to make it fit, however, the MIN_ADJUSTED_SCALE part doesn't make sense to me. It acts to clamp the scale to be <= 6, but without this having an impact on the output precision, which seems to be incorrect? Say I the computed output had a scale of 10 and a precision of 40, adjustScaleIfNeeded would then clamp this to a scale of 6 and a precision of 38. I would have thought it would either clamp to a scale of 8 and precision of 38, or a scale of 6 and a precision of 36? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
