Jo2234 opened a new pull request, #11105: URL: https://github.com/apache/arrow-rs/pull/11105
# Which issue does this PR close? Related to #6882 and the review of #10361. This is the separate schema-inference change requested in that review; it does not implement header detection or close #6882. # Rationale for this change CSV numeric parsers accept a leading plus sign, but schema inference currently classifies `+1` and `+1.5` as UTF-8. A column containing these values can therefore be inferred as text even when the same input decodes with an explicit numeric schema. # What changes are included in this PR? Allow one optional `+` or `-` in the existing integer and decimal/scientific-notation inference regexes. Preserve the existing integer overflow fallback and rejection of malformed signs. No header-detection API or other inference rules are changed. # Are these changes tested? The regression covers signed integer, decimal and scientific notation, the signed i64 maximum, overflow to UTF-8, and malformed signs. It also decodes each input using the inferred schema and checks that every sampled row is retained. Validated on `984be43ee` against upstream `c60ac3fc6` with the repository-pinned Rust 1.98.1: - `cargo test -p arrow-csv --all-features`: 86 unit tests and 13 doctests passed. - `cargo test -p arrow --features csv`: 159 unit/integration tests and 11 doctests passed. - `cargo clippy -p arrow-csv --all-targets --all-features -- -D warnings`: passed. - `cargo fmt --all -- --check` and `git diff upstream/main --check`: passed. # Are there any user-facing changes? Yes. CSV columns with supported leading-plus numeric values can now infer as `Int64` or `Float64` instead of `Utf8`. Applications needing text can continue to supply an explicit schema. Oversized integers remain text. # AI assistance disclosure AI assistance was used for the extracted fix, regression tests, and this description. The diff was inspected against current upstream and the listed checks were run locally. Maintainer acceptance is pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
