[ https://issues.apache.org/jira/browse/SPARK-23604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated SPARK-23604: ----------------------------------- Description: We ran into an issue with a downstream build of Spark running against a custom Parquet build where {{ParquetInteroperabilityTestSuite}} started failing because {{Statistics.isEmpty}} changed its behavior as of PARQUET-1217. ({{isEmpty()}} now considers whether there are 0 or more nulls, and by default {{num_nulls}} is 0 for 'empty' stats objects). The test really cares about whether the statistics object has values, so a very simple fix to use {{hasNonNullValue}} instead corrects the issue. Filing it now because it's a backwards-compatible fix to the current Parquet version so we can fix it right now before we hit the issue in the future. was: We ran into an issue with a downstream build of Spark running against a custom Parquet build where {{ParquetInteroperabilityTestSuite}} started failing because {{Statistics.isEmpty}} changed its behavior as of PARQUET-1217. ({{isEmpty() now considers whether there are 0 or more nulls, and by default {{num_nulls}} is 0 for 'empty' stats objects). The test really cares about whether the statistics object has values, so a very simple fix to use {{hasNonNullValue}} instead corrects the issue. Filing it now because it's a backwards-compatible fix to the current Parquet version so we can fix it right now before we hit the issue in the future. > ParquetInteroperabilityTest timestamp test should use > Statistics.hasNonNullValue > -------------------------------------------------------------------------------- > > Key: SPARK-23604 > URL: https://issues.apache.org/jira/browse/SPARK-23604 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.3.0 > Reporter: Henry Robinson > Priority: Minor > > We ran into an issue with a downstream build of Spark running against a > custom Parquet build where {{ParquetInteroperabilityTestSuite}} started > failing because {{Statistics.isEmpty}} changed its behavior as of > PARQUET-1217. ({{isEmpty()}} now considers whether there are 0 or more nulls, > and by default {{num_nulls}} is 0 for 'empty' stats objects). > The test really cares about whether the statistics object has values, so a > very simple fix to use {{hasNonNullValue}} instead corrects the issue. Filing > it now because it's a backwards-compatible fix to the current Parquet version > so we can fix it right now before we hit the issue in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org