hanke580 opened a new issue, #54535:
URL: https://github.com/apache/arrow/issues/54535
### Describe the bug, including details regarding any error messages,
version, and platform.
The CSV writer emits a null cell as the unquoted `null_string`, which is
empty by default. In a table with a single column, a null row therefore becomes
an empty line. Arrow's CSV readers skip empty lines by default
(`ParseOptions.ignore_empty_lines = True`), so writing such a table with
default options and reading it back silently drops every null row.
`ds.write_dataset(..., format="csv")` and `ds.dataset(..., format="csv")`
behave the same.
**Reproducer**
```python
import io
import pyarrow as pa
import pyarrow.csv as csv
t = pa.table({"c": pa.array([1, None, 3, None, 5], pa.int64())})
buf = io.BytesIO()
csv.write_csv(t, buf)
print(buf.getvalue())
back = csv.read_csv(io.BytesIO(buf.getvalue()))
print(back.num_rows, back["c"].to_pylist()) # expected 5 [1, None, 3,
None, 5]
# string column: an empty string is quoted, a null is not
s = pa.table({"s": pa.array(["a", None, "", "b"])})
buf = io.BytesIO()
csv.write_csv(s, buf)
print(buf.getvalue(),
csv.read_csv(io.BytesIO(buf.getvalue()))["s"].to_pylist())
```
Output on main:
```
b'"c"\n1\n\n3\n\n5\n'
3 [1, 3, 5]
b'"s"\n"a"\n\n""\n"b"\n' ['a', '', 'b']
```
The file has 5 data rows but reads back as 3; the rows no longer line up
with their positions, and no error or warning is raised. The other options
behave as follows:
- `quoting_style="all_valid"` and `"none"` give the same result.
- With `include_header=False`, a leading null becomes an empty first line.
- Reading with `ParseOptions(ignore_empty_lines=False)` recovers all 5 rows.
- pandas `read_csv` (default `skip_blank_lines=True`) also reads 3 rows.
- Polars and DuckDB happen to read 5.
For comparison, Python's `csv` module writes a lone empty field as `""`
precisely so that the row is not an empty line
(`csv.writer(f).writerow([None])` writes `""`).
**Expected behavior:** a table written by `write_csv` with default options
reads back with the same rows from `read_csv` with default options. For
example, the writer could emit a null in a row with a single column as a quoted
empty string (`""`) or as some other non-empty marker, like the Python `csv`
module does.
For non-string columns, quoting is lossless: with default reader options a
quoted `""` already reads back as null
(`ConvertOptions.quoted_strings_can_be_null = True`), e.g.
`csv.read_csv(io.BytesIO(b'"c"\n1\n""\n3\n'))` gives `[1, None, 3]`. For string
columns, quoting would make the null read back as an empty string, so the
writer may prefer another marker there. Either way, it should not emit an empty
line that the default reader ignores.
**Versions:** pyarrow 27.0.0.dev13 (nightly built from main at 9f03413),
Python 3.12, Linux x86_64 (Ubuntu 22.04). The released pyarrow 21.0.0 through
26.0.0 wheels behave the same.
**Where it comes from**
- Both column populators copy `null_string_` followed by the end characters
for a null, without quoting and regardless of the number of columns
([csv/writer.cc#L277-L286](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/arrow/csv/writer.cc#L277-L286)
for `UnquotedColumnPopulator`,
[csv/writer.cc#L413-L421](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/arrow/csv/writer.cc#L413-L421)
for `QuotedColumnPopulator`).
- The reader's default is `ignore_empty_lines = true`. The option's own
comment says that only with `false` does "an empty line represent a single
empty value (assuming a one-column CSV file)"
([csv/options.h#L59-L61](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/arrow/csv/options.h#L59-L61)).
**Impact:** exporting and re-importing one-column data with pyarrow's
default CSV options is common, for example a list of ids or a single selected
column. Doing so silently loses every null row, which changes row counts and
breaks positional alignment with other data. Found by differential testing of
CSV round trips.
### Component(s)
C++, Python
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
