thisisnic commented on issue #20253:
URL: https://github.com/apache/arrow/issues/20253#issuecomment-6740201987
Still reproduces, needs more investigation, AI analysis below but requires
verification
----------------------------------------------
Still reproduces on 26.0.1 with the attached `test.json`: default options
fail about half the time, `use_threads=False` fails every
time. The error is always `ArrowNotImplementedError: JSON conversion to
struct<...> is not supported`; it never produces a table with wrong column
types.
The `timestamp[s]` types in the message are a red herring (strings are
inferred as timestamp first and widened to utf8 later). The
actual cause is in cross-block type promotion. When a block's conversion
fails, `chunked_builder.cc` calls `Promote()` on the leaf
builder. For a column whose type was fixed as null by an earlier block,
`Promote()` falls through to `Infer()`, which returns a
struct or list type if the value in the new block is an object or array.
That type is then passed to `MakeConverter()`, which has no
struct or list case and returns NotImplemented. Which block fixes the type
first depends on block boundaries and thread scheduling,
hence the sporadic behaviour.
Fix needed: a null-typed leaf builder promoted to a nested type should get
a nested chunked builder instead of a leaf converter.
Regression test: a column that is null in block 1 and an object in block
2. Retitled to reflect this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
