Joris Van den Bossche created ARROW-9476: --------------------------------------------
Summary: [C++][Dataset] HivePartitioning discovery with dictionary types fails for multiple fields Key: ARROW-9476 URL: https://issues.apache.org/jira/browse/ARROW-9476 Project: Apache Arrow Issue Type: Bug Components: C++ Reporter: Joris Van den Bossche Apparently, ARROW-9288 was not fully / correctly fixing the issue. With a single string partition field, it now works fine. But once you have multiple string fields, you get parsing errors. A reproducible example: {code} import numpy as np import pyarrow as pa import pyarrow.parquet as pq import pyarrow.dataset as ds foo_keys = np.array(['a', 'b', 'c'], dtype=object) bar_keys = np.array(['d', 'e', 'f'], dtype=object) N = 30 table = pa.table({ 'foo': foo_keys.repeat(10), 'bar': np.tile(np.tile(bar_keys, 5), 2), 'values': np.random.randn(N) }) base_path = "test_partition_directories3" pq.write_to_dataset(table, base_path, partition_cols=["bar", "foo"]) # works ds.dataset(base_path, partitioning="hive") # fails part = ds.HivePartitioning.discover(max_partition_dictionary_size=-1) ds.dataset(base_path, partitioning=part) {code} cc [~bkietz] -- This message was sent by Atlassian Jira (v8.3.4#803005)