pstrzelczak commented on issue #46228:
URL: https://github.com/apache/arrow/issues/46228#issuecomment-3048118009
I managed to work around the lack of this feature to infer the schema
reliably in a single pass by writing and reading in-memory csv file for each
record batch:
```
schemas = []
with pyarrow.csv.open_csv(csv_path) as input_csv:
for batch in input_csv:
buffer = io.BytesIO()
pyarrow.csv.write_csv(batch, buffer)
buffer.seek(0)
inferred_table = pyarrow.csv.read_csv(buffer)
current_chunk_schema = inferred_table.schema
if current_chunk_schema not in schemas:
schemas.append(current_chunk_schema)
```
and then resolve multiple schemas using pyarrow.unify_schemas
Still, it would be great to avoid additional steps and use (not available
now) the option to infer schema for each individual chunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
