goatsweater commented on issue #40107:
URL: https://github.com/apache/arrow/issues/40107#issuecomment-2286921061
An elaboration of the test above shows inconsistent behavior when providing
a defined schema too.
Multi-record files:
```python
import tempfile
import pathlib
import pyarrow as pa
import pyarrow.dataset as ds
base = pathlib.Path(tempfile.mkdtemp(prefix="pyarrow-"))
(base / "json_dataset").mkdir(exist_ok=True)
schema = pa.schema([
pa.field("userId", pa.string())
])
# Make file 1
f = open(base / "json_dataset/data1.json", "a")
f.write('{"userId": null}\n{"userId": null}')
f.close()
# Make file 2
f = open(base / "json_dataset/data2.json", "a")
f.write('{"userId": null}\n{"userId": "example-string"}')
f.close()
# writing it into two parquet files
table = ds.dataset(base / "json_dataset", format="json", schema=schema)
print(table.to_table())
```
```
pyarrow.Table
userId: string
----
userId: [[null,null],[null,"example-string"]]
```
vs single record files:
```python
import tempfile
import pathlib
import pyarrow as pa
import pyarrow.dataset as ds
base = pathlib.Path(tempfile.mkdtemp(prefix="pyarrow-"))
(base / "json_dataset").mkdir(exist_ok=True)
schema = pa.schema([
pa.field("userId", pa.string())
])
# Make file 1
f = open(base / "json_dataset/data1.json", "a")
f.write('{"userId": null}')
f.close()
# Make file 2
f = open(base / "json_dataset/data2.json", "a")
f.write('{"userId": "example-string"}')
f.close()
# writing it into two parquet files
table = ds.dataset(base / "json_dataset", format="json", schema=schema)
print(table.to_table())
```
```
pyarrow.Table
userId: string
----
userId: [[null],[null]]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
