[ https://issues.apache.org/jira/browse/ARROW-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joris Van den Bossche resolved ARROW-7702. ------------------------------------------ Fix Version/s: 1.0.0 Assignee: Francois Saint-Jacques Resolution: Duplicate > [C++][Dataset] Provide (optional) deterministic order of batches > ---------------------------------------------------------------- > > Key: ARROW-7702 > URL: https://issues.apache.org/jira/browse/ARROW-7702 > Project: Apache Arrow > Issue Type: Bug > Components: C++, Python > Reporter: Joris Van den Bossche > Assignee: Francois Saint-Jacques > Priority: Major > Labels: dataset > Fix For: 1.0.0 > > > Example with python: > {code} > import pyarrow as pa > import pyarrow.parquet as pq > table = pa.table({'a': range(12)}) > pq.write_table(table, "test_chunks.parquet", chunk_size=3) > # reading with dataset > import pyarrow.dataset as ds > ds.dataset("test_chunks.parquet").to_table().to_pandas() > {code} > gives non-deterministic result (order of the row groups in the parquet file): > {code} > In [25]: ds.dataset("test_chunks.parquet").to_table().to_pandas() > > > Out[25]: > a > 0 0 > 1 1 > 2 2 > 3 3 > 4 4 > 5 5 > 6 6 > 7 7 > 8 8 > 9 9 > 10 10 > 11 11 > In [26]: ds.dataset("test_chunks.parquet").to_table().to_pandas() > > > Out[26]: > a > 0 0 > 1 1 > 2 2 > 3 3 > 4 8 > 5 9 > 6 10 > 7 11 > 8 4 > 9 5 > 10 6 > 11 7 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)