GujaLomsadze opened a new pull request, #50834:
URL: https://github.com/apache/arrow/pull/50834
### Rationale for this change
Closes #46918
`add_column` / `set_column` on both `Table` and `RecordBatch` convert the
column
data before they look at `field_`. Column data that is entirely null is
therefore
inferred as the null type, and `AddColumn` / `SetColumn` then reject it
against an
explicitly typed `Field`:
```python
>>> t = pa.table({"A": [1.0, 2.0]})
>>> t.append_column(pa.field("B", pa.float64(), nullable=True), [[None,
None]])
ArrowInvalid: Field type did not match data type
```
The type the user asked for was available in `field_` the whole time, it
just was
not used for the conversion.
### What changes are included in this PR?
When `field_` is a `Field` and the converted column came out as null type
while the
field type is not null, the column is cast to the field type. Four call
sites, same
two lines each:
- `Table.add_column`
- `Table.set_column`
- `RecordBatch.add_column`
- `RecordBatch.set_column`
`append_column` on both classes forwards to `add_column`, so it is covered
too.
The issue only reports `append_column`, but `set_column` has the identical
defect,
so it is fixed here as well. Happy to split it out if reviewers prefer.
Data that is not all-null keeps its inferred type, so a genuine mismatch is
still
reported rather than silently cast:
```python
>>> t.append_column(pa.field("B", pa.int64()), [[1.5, 2.5]])
ArrowInvalid: Field type did not match data type # unchanged
```
This narrow scope is deliberate. Always converting with the field type would
be more
uniform with `pa.array(data, type=...)`, but that API truncates silently
(`pa.array([1.5, 2.5], type=pa.int64())` gives `[1, 2]`), so the broad
version would
turn today's loud error into a silent data change. Happy to switch to the
broad
behaviour if that is preferred.
### Are these changes tested?
Yes. `test_table_add_column_all_null_data_typed_field` (parametrized over
`Table`/`RecordBatch` x `add_column`/`set_column`) and
`test_table_append_column_all_null_data_typed_field`. They cover the typed
field
case, a null field with null data staying null, and a non-null mismatch still
raising.
### Are there any user-facing changes?
Yes, a bug fix. Calls that raised `ArrowInvalid` / `ArrowTypeError` now
succeed and
produce a column of the requested type. No previously working call changes
behaviour.
### Note
One pre-existing inconsistency noticed while writing the tests, left alone
here:
for the same type mismatch `Table` raises `ArrowInvalid` while `RecordBatch`
raises
`ArrowTypeError`. Can file a separate issue if that is worth aligning.
### AI usage disclosure
This change was written with the assistance of an AI coding agent (Claude
Code).
The agent located the four call sites, wrote the patch and the tests, and
ran the
test suite. I reviewed the diff, chose the narrow-vs-broad scope, and
verified the
results before submitting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
