dentiny opened a new pull request, #11695:
URL: https://github.com/apache/arrow-rs/pull/11695
# Which issue does this PR close?
No existing issue.
# Rationale for this change
Some JSON Lines producers emit bare `NaN`, `Infinity`, and `-Infinity`, for
example `{"metadata":{"year":NaN}}`. Arrow can represent these values in
floating-point arrays, but the JSON tokenizer currently rejects them before
type-specific decoding. An opt-in extension lets known-schema readers ingest
these sources without rewriting input or converting values to null.
# What changes are included in this PR?
- Add `ReaderBuilder::with_allow_non_finite_floats`, disabled by default to
retain standard JSON behavior.
- Recognize the three literals in the existing streaming tokenizer, using
its literal states and numeric tape entries. This reuses existing float
decoding, primitive-to-string coercion, and type-conflict handling, including
nested fields and lists.
- Handle `-Infinity` when the minus sign and the rest of the literal arrive
in different buffers.
# Are these changes tested?
Two focused tests cover Float16/32/64 values and null validity, default
rejection, incompatible integer columns, primitive-to-string coercion, nested
Struct/List values, unchanged strings, malformed/truncated input, and every
input chunk size for a nested record. The existing Arrow JSON tests and
doctests also pass (162 total). Changed-package Clippy checks pass; including
path dependencies exposes existing `arrow-data` `from_iter_instead_of_collect`
lint failures, which this PR does not modify.
A local check with a real source record confirms a valid Float64 NaN and
unchanged text without input rewriting.
# Are there any significant user-facing changes?
This is an additive opt-in reading API; existing readers retain their
behavior. Schema inference APIs remain unchanged and do not gain support for
nonstandard literals in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
