Mr-Nobody-Anonymous opened a new pull request, #54691:
URL: https://github.com/apache/arrow/pull/54691
### Rationale for this change
In #50926, pad_short_rows was implemented in the C++ CSV parser/reader
(rrow::csv::ParseOptions::pad_short_rows), allowing rows with fewer fields
than expected to be padded with null values rather than failing. However, this
option was not yet exposed to the Python bindings (#53290).
### What changes are included in this PR?
- Expose pad_short_rows on CCSVParseOptions in
python/pyarrow/includes/libarrow.pxd.
- Add pad_short_rows parameter to pyarrow.csv.ParseOptions.__init__,
defaulting to None (preserving C++ default alse).
- Add pad_short_rows property (getter and setter) on
pyarrow.csv.ParseOptions.
- Update ParseOptions.equals(), _repr_base(), __getstate__(), and
backwards-compatible __setstate__().
- Document ~ParseOptions.pad_short_rows in docs/source/python/csv.rst.
- Add unit tests in python/pyarrow/tests/test_csv.py:
- Options validation, string/repr output, and serialization/pickling.
- CSV table and streaming reading with pad_short_rows=True across serial
and parallel readers.
- Verification that default pad_short_rows=False retains strict error
behavior.
- Reading with explicit column names and typed schema conversions.
- Add test coverage in python/pyarrow/tests/test_dataset.py for
ds.CsvFileFormat(pa.csv.ParseOptions(pad_short_rows=True)).
### Are these changes tested?
Yes. Added tests in est_csv.py and est_dataset.py covering parse options
validation, pickling, repr, and reading short rows in serial and parallel modes.
### Are there any user-facing changes?
Yes. pyarrow.csv.ParseOptions now accepts pad_short_rows (defaulting to
False), allowing users to opt into padding short rows with nulls.
### Was AI used for this PR?
AI assistance was used for initial research and drafting. The changes,
tests, and documentation were reviewed and verified.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #53290
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
