[ https://issues.apache.org/jira/browse/PARQUET-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wes McKinney resolved PARQUET-1369. ----------------------------------- Resolution: Fixed Fix Version/s: (was: cpp-1.5.0) 1.10.1 Issue resolved by pull request 2655 [https://github.com/apache/arrow/pull/2655] > [Python] Unavailable Parquet column statistics from Spark-generated file > ------------------------------------------------------------------------ > > Key: PARQUET-1369 > URL: https://issues.apache.org/jira/browse/PARQUET-1369 > Project: Parquet > Issue Type: Bug > Components: parquet-cpp > Affects Versions: cpp-1.4.0 > Reporter: Robert Gruener > Assignee: Robert Gruener > Priority: Major > Labels: parquet, pull-request-available > Fix For: 1.10.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > I have a dataset generated by spark which shows it has statistics for the > string column when using the java parquet-mr code (shown by using > `parquet-tools meta`) however reading from pyarrow shows that the statistics > for that column are not set. I should not the column only has a single > value, though it still seems like a problem that pyarrow can't recognize it > (it can recognize statistics set for the long and double types). > See https://github.com/apache/arrow/files/2161147/metadata.zip for file > example. > Pyarrow Code To Check Statistics: > {code} > from pyarrow import parquet as pq > meta = pq.read_metadata('/tmp/metadata.parquet') > # No Statistics For String Column, prints false and statistics object is None > print(meta.row_group(0).column(1).is_stats_set) > {code} > Example parquet-meta output: > {code} > file schema: spark_schema > -------------------------------------------------------------------------------- > int: REQUIRED INT64 R:0 D:0 > string: OPTIONAL BINARY O:UTF8 R:0 D:1 > float: REQUIRED DOUBLE R:0 D:0 > row group 1: RC:8333 TS:76031 OFFSET:4 > -------------------------------------------------------------------------------- > int: INT64 SNAPPY DO:0 FPO:4 SZ:7793/8181/1.05 VC:8333 > ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0, max: 100, num_nulls: 0] > string: BINARY SNAPPY DO:0 FPO:7797 SZ:1146/1139/0.99 VC:8333 > ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE ST:[min: hello, max: hello, num_nulls: > 4192] > float: DOUBLE SNAPPY DO:0 FPO:8943 SZ:66720/66711/1.00 VC:8333 > ENC:PLAIN,BIT_PACKED ST:[min: 0.0057611096964338415, max: 99.99811053829232, > num_nulls: 0] > {code} > I realize the column only has a single value though it still seems like > pyarrow should be able to read the statistics set. I made this here and not a > JIRA since I wanted to be sure this is actually an issue and there wasnt a > ticket already made there (I couldnt find one but I wanted to be sure). > Either way I would like to understand why this is -- This message was sent by Atlassian JIRA (v7.6.3#76005)