[ https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16817308#comment-16817308 ]
Pearu Peterson commented on ARROW-1983: --------------------------------------- Currently, ParquetDataset metadata has the following approximate data structure (type-specs are shown only for the relevant attributes): {noformat} ParquetDataset: list<ParquetDatasetPiece> pieces list<str> paths fs common_metadata, common_metadata_path metadata, metadata_path ParquetDatasetPiece: sting path get_metadata() -> FileMetaData partition_keys FileMetaData: list<RowGroupMetaData> row_groups ParquetSchema schema dict metadata = {b‘pandas’: <pandas metadata in bytes>} int num_rows, num_columns str format_version, created_by RowGroupMetaData: list<ColumnChunkMetaData> columns int num_rows, total_byte_size ColumnChunkMetaData: str physical_type, encodings, path_in_schema, compression int num_values, total_uncompressed_size, total_compressed_size, data_page_offset, index_page_offset, dictionary_page_offset RowGroupStatistics statistics RowGroupStatistics: bool has_min_max int min, max, null_count, distinct_count, num_values str physical_type{noformat} If only the data in RowGroupStatistics is relevant for this issue (please confirm), then the statistics data could be collected into a single Parquet file, say `_statistics`, containing the following columns: {noformat} <Piece.path>, <RowGroup index>, <ColumnChunk index>, <RowGroupStatistics data>{noformat} [~mrocklin], would the information in `_statistics` sufficient for Dask needs? > [Python] Add ability to write parquet `_metadata` file > ------------------------------------------------------ > > Key: ARROW-1983 > URL: https://issues.apache.org/jira/browse/ARROW-1983 > Project: Apache Arrow > Issue Type: Improvement > Components: C++, Python > Reporter: Jim Crist > Priority: Major > Labels: beginner, parquet > Fix For: 0.14.0 > > > Currently {{pyarrow.parquet}} can only write the {{_common_metadata}} file > (mostly just schema information). It would be useful to add the ability to > write a {{_metadata}} file as well. This should include information about > each row group in the dataset, including summary statistics. Having this > summary file would allow filtering of row groups without needing to access > each file beforehand. > This would require that the user is able to get the written RowGroups out of > a {{pyarrow.parquet.write_table}} call and then give these objects as a list > to new function that then passes them on as C++ objects to {{parquet-cpp}} > that generates the respective {{_metadata}} file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)