andygrove commented on PR #4971:
URL:
https://github.com/apache/datafusion-comet/pull/4971#issuecomment-5876429669
This is a light fully automated review since there are so many PRs open.
Selected floats come back in a different notation from Spark. This predates
the PR, but it now lives in the new `PathResult::write`. Spark's generator
copies a float token through `double` (Jackson's `_copyCurrentFloatValue` calls
`writeNumber(p.getDoubleValue())`, which prints with `Double.toString`), while
`value.to_string()` at
`native/spark-expr/src/string_funcs/get_json_object.rs:510` and the flatten
leaves at line 559 use serde_json's shortest form. So
`get_json_object('{"a":0.0001}', '$.a')` returns `1.0E-4` in Spark and `0.0001`
natively, and `12345678.9` comes back from Spark as `1.23456789E7`. Floats
nested inside a returned object or array differ the same way, since
`copyCurrentStructure` copies them through the same method. Unlike the
1,000-digit cases, these are ordinary values. Neither this nor the "selected
very large numeric values" gap from the description is in
`getIncompatibleReasons()` at
`spark/src/main/scala/org/apache/comet/serde/strings.scala:699`, which is
what the generated compatibility page is built from. Could the output go
through a `serde_json::ser::Formatter` whose `write_f64` uses the existing
`write_java_float_string` in
`native/spark-expr/src/conversion_funcs/numeric.rs`, with a few of these values
added to `get_json_object.sql`? If that is more than this PR should take on, an
issue plus a reasons entry and these queries behind `ignore(<issue link>)`
would at least record the gap where the next person will look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
