[
https://issues.apache.org/jira/browse/SPARK-23604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-23604.
------------------------------------
Resolution: Fixed
Fix Version/s: 2.4.0
Issue resolved by pull request 20740
[https://github.com/apache/spark/pull/20740]
> ParquetInteroperabilityTest timestamp test should use
> Statistics.hasNonNullValue
> --------------------------------------------------------------------------------
>
> Key: SPARK-23604
> URL: https://issues.apache.org/jira/browse/SPARK-23604
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Priority: Minor
> Fix For: 2.4.0
>
>
> We ran into an issue with a downstream build of Spark running against a
> custom Parquet build where {{ParquetInteroperabilityTestSuite}} started
> failing because {{Statistics.isEmpty}} changed its behavior as of
> PARQUET-1217. ({{isEmpty()}} now considers whether there are 0 or more nulls,
> and by default {{num_nulls}} is 0 for 'empty' stats objects).
> The test really cares about whether the statistics object has values, so a
> very simple fix to use {{hasNonNullValue}} instead corrects the issue. Filing
> it now because it's a backwards-compatible fix to the current Parquet version
> so we can fix it right now before we hit the issue in the future.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
