AndreaBozzo opened a new issue, #10578:
URL: https://github.com/apache/arrow-rs/issues/10578
### Is your feature request related to a problem or challenge?
`ReaderBuilder::with_truncated_rows(true)` recovers a row that has fewer
fields
than the schema by padding it. There is no equivalent for a row that has more
fields than the schema. That case always fails:
```
Csv error: incorrect number of fields for line 3, expected 3 got 4
```
The flag is documented as controlling truncated rows, so the behaviour is
consistent with its name. The gap is that a caller who wants to tolerate
ragged
input has no way to express that for both directions. Most CSV readers treat
this as a single setting: the `csv` crate has `flexible`, which accepts
records
of any length, and pandas has `on_bad_lines`.
Repro against arrow-csv 59.1.0, with the flag in both positions:
```rust
use arrow_csv::ReaderBuilder;
use arrow_schema::{DataType, Field, Schema};
use std::io::Cursor;
use std::sync::Arc;
fn main() {
let schema = Arc::new(Schema::new(vec![
Field::new("name", DataType::Utf8, true),
Field::new("age", DataType::Utf8, true),
Field::new("city", DataType::Utf8, true),
]));
let csv = "name,age,city\nAlice,25,Rome\nBob,30,Milan,EXTRA\n";
for allow in [true, false] {
let reader = ReaderBuilder::new(schema.clone())
.with_header(true)
.with_truncated_rows(allow)
.build(Cursor::new(csv))
.unwrap();
// Both settings error on line 3.
println!("with_truncated_rows({allow}): {:?}",
reader.into_iter().next().unwrap().err());
}
}
```
### Describe the solution you'd like
An opt-in that discards fields past the schema width, so both directions of
raggedness can be recovered by configuration.
Naming is the open question, and I do not have a strong preference:
- `with_ragged_rows(bool)`, covering both directions in one flag, closest to
the
`csv` crate's `flexible`.
- An enum such as `ExtraFields::{Error, Ignore}`, if the two directions are
worth controlling independently.
The second is more flexible and keeps `with_truncated_rows` meaning what it
says
today. The first is simpler and matches what callers usually want.
### Describe alternatives you've considered
Pre-scanning the input with the `csv` crate to find the widest record,
building
the Arrow schema at that width, then projecting the surplus columns away
after
decoding. This is what we shipped. It works, but it costs a second full read
of
the input and produces a schema that does not describe the data, purely so
that
the decoder has somewhere to put fields that are going to be discarded.
Rejecting the file outright was the other option. We chose against it because
the same input parses without complaint through our non-Arrow engine, so the
choice of engine would have changed whether a file was readable.
### Additional context
Raised separately from #10577 at a maintainer's suggestion. The two are
independent: #10577 asks for a count of rows already being repaired, which is
additive and cheap, while this one changes what the decoder accepts and is
the
larger change of the two.
Context for why both directions matter to us: dataprof is a data profiler,
and
a row with a field count that disagrees with the header is a structural
violation we report on regardless of which direction it disagrees in. Our
Arrow-backed CSV engine was the only one that rejected one direction while
silently repairing the other
([dataprof#470](https://github.com/AndreaBozzo/dataprof/issues/470)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
