[
https://issues.apache.org/jira/browse/ARROW-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457117#comment-17457117
]
Adrien Hoarau commented on ARROW-14701:
---------------------------------------
> It does seem to me to be based on rows; what makes you say otherwise?
I had this impression, but checking again, it does seem to be based on the
number or rows with the threshold you suggested, so my mistake.
> [Python] parquet.write_table has an undocumented and silent cap on
> row_group_size
> ---------------------------------------------------------------------------------
>
> Key: ARROW-14701
> URL: https://issues.apache.org/jira/browse/ARROW-14701
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 6.0.0
> Reporter: Adrien Hoarau
> Priority: Minor
>
> {code:java}
> from io import BytesIO
> import pandas as pd
> import pyarrow
> from pyarrow import parquet
> from pyarrow import fs
> print(pyarrow._version_)
> def check_row_groups_created(size: int):
> df = pd.DataFrame({"a": range(size)})
> t = pyarrow.Table.from_pandas(df)
> buffer = BytesIO()
> parquet.write_table(t, buffer, row_group_size=size)
> buffer.seek(0)
> print(parquet.read_metadata(buffer))
>
> check_row_groups_created(50_000_000)
> check_row_groups_created(100_000_000) {code}
> outputs:
> {code:java}
> 6.0.0
> <pyarrow._parquet.FileMetaData object at 0x7f838584ab80>
> created_by: parquet-cpp-arrow version 6.0.0
> num_columns: 1
> num_rows: 50000000
> num_row_groups: 1
> format_version: 1.0
> serialized_size: 1493
> <pyarrow._parquet.FileMetaData object at 0x7f838584ab80>
> created_by: parquet-cpp-arrow version 6.0.0
> num_columns: 1
> num_rows: 100000000
> num_row_groups: 2
> format_version: 1.0
> serialized_size: 1640 {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
