viirya commented on issue #2387: URL: https://github.com/apache/arrow-rs/issues/2387#issuecomment-1209661359
> What is the purpose of the precision argument? Is it just for interoperability with other non-arrow representations? > Is there a requirement to saturate/error at the bounds of the precision, or can we simply overflow/saturate at the bounds of the underlying representation > Does validating the precision on ingest to ArrayData actually elide any validation when performing computation? I think that the precision for decimal specifies the representation range like other types do. It's useful so we can know at which moment an overflow can be happened and what range of values are possible to use with the type without overflow. As we don't have native decimal type in Rust, it is internally represented by bytes with maximum length for the maximum representation range. It could be easily specified with a value over its precision. For example, we can create a 128-bit decimal with precision 5 by giving it a value which is represent-able only by precision 10 (i.e., 10 digits). For native types, you might get a overlfow value, but for this case the 10-digits value still can be put into 128-bit bytes. We don't have it as overflow value there, I think. I think that is why the things become complicated. We may need to validate decimal values at some moments. For now we throw some errors for invalid (overflow) values. Instead, I think it might also make sense to change the invalid values to overflow representation to simulate overflow. Although I think for some systems, overflow will throw error anyway. But as mentioned: > As far as I can tell we don't protect against overflow of normal integer types I guess that decimal overflow should be no different. I think that interoperability is a concern. Because I'm not sure how other systems will interpret the decimal values from this crate. Assume we don't do any validation on decimals. If they just take the values without validation, I think the behavior might be unexpected. It could be an overflow at that system, or it truncates the value? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
