[jira] [Created] (ARROW-12066) [Python] Dataset API seg fault when filtering string column for None
Thomas Blauth created ARROW-12066:
-
Summary: [Python] Dataset API seg fault when filtering string
column for None
Key: ARROW-12066
URL: https://issues.apache.org/jira/browse/ARROW-12066
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 3.0.0
Environment: macOS 10.15.7
Reporter: Thomas Blauth
Trying to load a parquet file using the dataset api leads to a segmentation
fault when filtering string columns for None values.
Minimal reproducing example:
{code:python}
import pyarrow as pa
import pyarrow.dataset
import pyarrow.parquet
import pandas as pd
path = "./test.parquet"
df = pd.DataFrame({"A": ("a", "b", None)})
pa.parquet.write_table(pa.table(df), path)
ds = pa.dataset.dataset(path, format="parquet")
filter = pa.dataset.field("A") == pa.dataset.scalar(None)
table = ds.to_table(filter=filter)
{code}
Backtrace:
{code:bash}
(lldb) target create "/usr/local/mambaforge/envs/xxx/bin/python"
Current executable set to '/usr/local/mambaforge/envs/xxx/bin/python' (x86_64).
(lldb) settings set -- target.run-args "./tmp.py"
(lldb) r
Process 35235 launched: '/usr/local/mambaforge/envs/xxx/bin/python' (x86_64)
Process 35235 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS
(code=1, address=0x9)
frame #0: 0x00010314be48 libarrow.300.0.0.dylib`arrow::Status
arrow::VisitScalarInline(arrow::Scalar const&,
arrow::ScalarHashImpl*) + 104
libarrow.300.0.0.dylib`arrow::VisitScalarInline:
-> 0x10314be48 <+104>: cmpb $0x0, 0x9(%rax)
0x10314be4c <+108>: je 0x10314c0bc ; <+732>
0x10314be52 <+114>: movq 0x10(%rax), %rdi
0x10314be56 <+118>: movq 0x20(%rax), %rsi
Target 0: (python) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS
(code=1, address=0x9)
* frame #0: 0x00010314be48 libarrow.300.0.0.dylib`arrow::Status
arrow::VisitScalarInline(arrow::Scalar const&,
arrow::ScalarHashImpl*) + 104
frame #1: 0x00010314bd4f
libarrow.300.0.0.dylib`arrow::ScalarHashImpl::AccumulateHashFrom(arrow::Scalar
const&) + 111
frame #2: 0x000103134bca
libarrow.300.0.0.dylib`arrow::Scalar::Hash::hash(arrow::Scalar const&) + 42
frame #3: 0x000132fa0ea8
libarrow_dataset.300.0.0.dylib`arrow::dataset::Expression::hash() const + 264
frame #4: 0x000132fc913c
libarrow_dataset.300.0.0.dylib`std::__1::__hash_const_iterator*> std::__1::__hash_table,
std::__1::allocator
>::find(arrow::dataset::Expression const&) const +
28
frame #5: 0x000132faca9b
libarrow_dataset.300.0.0.dylib`arrow::Result
arrow::dataset::Modify(arrow::dataset::Expression,
arrow::dataset::Canonicalize(arrow::dataset::Expression,
arrow::compute::ExecContext*)::$_1 const&,
arrow::dataset::Canonicalize(arrow::dataset::Expression,
arrow::compute::ExecContext*)::$_9 const&) + 123
frame #6: 0x000132fac623
libarrow_dataset.300.0.0.dylib`arrow::dataset::Canonicalize(arrow::dataset::Expression,
arrow::compute::ExecContext*) + 131
frame #7: 0x000132fac76d
libarrow_dataset.300.0.0.dylib`arrow::dataset::Canonicalize(arrow::dataset::Expression,
arrow::compute::ExecContext*) + 461
frame #8: 0x000132fb00cb
libarrow_dataset.300.0.0.dylib`arrow::dataset::SimplifyWithGuarantee(arrow::dataset::Expression,
arrow::dataset::Expression const&)::$_10::operator()() const + 75
frame #9: 0x000132faf6b5
libarrow_dataset.300.0.0.dylib`arrow::dataset::SimplifyWithGuarantee(arrow::dataset::Expression,
arrow::dataset::Expression const&) + 517
frame #10: 0x000132f893f8
libarrow_dataset.300.0.0.dylib`arrow::dataset::Dataset::GetFragments(arrow::dataset::Expression)
+ 88
frame #11: 0x000132f8d25c
libarrow_dataset.300.0.0.dylib`arrow::dataset::GetFragmentsFromDatasets(std::__1::vector,
std::__1::allocator > > const&,
arrow::dataset::Expression)::'lambda'(std::__1::shared_ptr)::operator()(std::__1::shared_ptr)
const + 76
frame #12: 0x000132f8cd6c
libarrow_dataset.300.0.0.dylib`arrow::MapIterator,
std::__1::allocator > > const&,
arrow::dataset::Expression)::'lambda'(std::__1::shared_ptr),
std::__1::shared_ptr,
arrow::Iterator > >::Next() + 316
frame #13: 0x000132f8cb27
libarrow_dataset.300.0.0.dylib`arrow::Result
> >
arrow::Iterator
>
>::Next,
std::__1::allocator > > const&,
arrow::dataset::Expression)::'lambda'(std::__1::shared_ptr),
std::__1::shared_ptr,
arrow::Iterator > > >(void*) + 39
frame #14: 0x000132f8dcdb
libarrow_dataset.300.0.0.dylib`arrow::Iterator
> >::Next() + 43
frame #15: 0x000132f8d692
libarrow_dataset.300.0.0.dylib`arrow::FlattenIterator
>::Next() + 258
frame #16: 0x000132f8d477
libarrow_dataset.300.0.0.dylib`arrow::Result
> arrow::Iterator
>::Next >
>(void*) + 39
frame #17: 0x000132f8de0b
libarrow_dataset.300.0.0.dylib`arrow::It
[jira] [Created] (ARROW-12336) [C++][Python] Empty Int64 array is of wrong size
Thomas Blauth created ARROW-12336:
-
Summary: [C++][Python] Empty Int64 array is of wrong size
Key: ARROW-12336
URL: https://issues.apache.org/jira/browse/ARROW-12336
Project: Apache Arrow
Issue Type: Bug
Components: C++, Python
Environment: macOS 10.15.7
Arrow version: 3.1.0.dev578
Reporter: Thomas Blauth
Setup:
Table with Int64 and str columns; generated using the dataset api; filtered on
str column.
Bug Description:
Calling {{table.to_pandas()}} fails due to an empty array of the ChunkedArray
of the Int64 column. This empty array has a size of 4 Byte when using the arrow
nightly builds and 0 Byte when using arrow 3.0.0.
Note: The bug does not occur when the table only contains an Int64 column.
Minimal example:
{code:python}
import pandas as pd
import pyarrow as pa
import pyarrow.parquet
import pyarrow.dataset
print("Arrow version: " + str(pa.__version__))
print("---")
# Only Int64 works fine
df = pd.DataFrame({"Int_col": [1, 2, 10]}, dtype="Int64")
table = pa.table(df)
path_0 = "./test_0.parquet"
pa.parquet.write_table(table, path_0)
schema = pa.parquet.read_schema(path_0)
ds = pa.dataset.FileSystemDataset.from_paths(
paths=[path_0],
filesystem=pa.fs.LocalFileSystem(),
schema=schema,
format=pa.dataset.ParquetFileFormat(),
)
table = ds.to_table(filter=(pa.dataset.field("Int_col") == 3))
print("Size of array: " + str(table.column(0).nbytes))
df = table.to_pandas()
print("---")
# Int64 and str crashes
df = pd.DataFrame({"Int_col": [1, 2, 10], "str_col": ["A", "B", "Z"]})
df = df.astype({"Int_col": "Int64"})
table = pa.table(df)
path_1 = "./test_1.parquet"
pa.parquet.write_table(table, path_1)
schema = pa.parquet.read_schema(path_1)
ds = pa.dataset.FileSystemDataset.from_paths(
paths=[path_1],
filesystem=pa.fs.LocalFileSystem(),
schema=schema,
format=pa.dataset.ParquetFileFormat(),
)
table = ds.to_table(filter=(pa.dataset.field("str_col") == "C"))
print("Size of array: " + str(table.column(0).nbytes))
df = table.to_pandas()
{code}
Output :
{code:bash}
Arrow version: 3.1.0.dev578
---
Size of array: 0
---
Size of array: 4
Traceback (most recent call last):
File "/Users/xxx/empty_array_buffer_size.py", line 47, in
df = table.to_pandas()
File "pyarrow/array.pxi", line 756, in
pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow/table.pxi", line 1740, in pyarrow.lib.Table._to_pandas
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pyarrow/pandas_compat.py",
line 794, in table_to_blockmanager
blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pyarrow/pandas_compat.py",
line 1135, in _table_to_blocks
return [_reconstruct_block(item, columns, extension_columns)
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pyarrow/pandas_compat.py",
line 1135, in
return [_reconstruct_block(item, columns, extension_columns)
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pyarrow/pandas_compat.py",
line 753, in _reconstruct_block
pd_ext_arr = pandas_dtype.__from_arrow__(arr)
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pandas/core/arrays/integer.py",
line 117, in __from_arrow__
data, mask = pyarrow_array_to_numpy_and_mask(arr, dtype=self.type)
File
"/usr/local/mambaforge/envs/pa_nightly/lib/python3.9/site-packages/pandas/core/arrays/_arrow_utils.py",
line 32, in pyarrow_array_to_numpy_and_mask
data = np.frombuffer(buflist[1], dtype=dtype)[arr.offset : arr.offset +
len(arr)]
ValueError: buffer size must be a multiple of element size
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
