Cheng Lian created PARQUET-136:
----------------------------------
Summary: NPE thrown in StatisticsFilter when all values in a
string/binary column trunk are null
Key: PARQUET-136
URL: https://issues.apache.org/jira/browse/PARQUET-136
Project: Parquet
Issue Type: Bug
Components: parquet-mr
Affects Versions: parquet-mr_1.6.0
Reporter: Cheng Lian
For a string or a binary column, if all values in a single column trunk are
null, so do the min & max values in the column trunk statistics. However, while
checking the statistics for column trunk pruning, a null check is missing, and
causes NPE. Corresponding code can be found
[here|https://github.com/apache/incubator-parquet-mr/blob/251a495d2a72de7e892ade7f64980f51f2fcc0dd/parquet-hadoop/src/main/java/parquet/filter2/statisticslevel/StatisticsFilter.java#L97-L100].
This issue can be steadily reproduced with the following Spark shell snippet
against Spark 1.2.0-SNAPSHOT
([013089794d|https://github.com/apache/spark/tree/013089794ddfffbae8b913b72c1fa6375774207a]):
{code}
import org.apache.spark.sql.SQLContext
val sqlContext = new SQLContext(sc)
import sqlContext._
case class StringCol(value: String)
sc.parallelize(StringCol(null) :: Nil,
1).saveAsParquetFile("/tmp/empty.parquet")
parquetFile("/tmp/empty.parquet").registerTempTable("null_table")
sql("SET spark.sql.parquet.filterPushdown=true")
sql("SELECT * FROM null_table WHERE value = 'foo'").collect()
{code}
Exception thrown:
{code}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID
0, localhost): java.lang.NullPointerException
at
parquet.io.api.Binary$ByteArrayBackedBinary.compareTo(Binary.java:206)
at
parquet.io.api.Binary$ByteArrayBackedBinary.compareTo(Binary.java:162)
at
parquet.filter2.statisticslevel.StatisticsFilter.visit(StatisticsFilter.java:100)
at
parquet.filter2.statisticslevel.StatisticsFilter.visit(StatisticsFilter.java:47)
at parquet.filter2.predicate.Operators$Eq.accept(Operators.java:162)
at
parquet.filter2.statisticslevel.StatisticsFilter.canDrop(StatisticsFilter.java:52)
at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:46)
at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:22)
at
parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:108)
at
parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:28)
at
parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:158)
at
parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:138)
at
org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:135)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:107)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
