Romi Kuntsman created SPARK-10135:
-------------------------------------
Summary: Percent of pruned partitions is shown wrong
Key: SPARK-10135
URL: https://issues.apache.org/jira/browse/SPARK-10135
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.4.0
Reporter: Romi Kuntsman
Priority: Trivial
When reading partitioned Parquet in SparkSQL, an info message about the number
of pruned partitions is displayed.
Actual:
"Selected 15 partitions out of 181, pruned -1106.6666666666667% partitions."
Expected:
"Selected 15 partitions out of 181, pruned 91.71270718232044% partitions."
Fix: (i'm newbie here so please help make patch, thanks!)
in DataSourceStrategy.scala in method apply()
insted of:
val percentPruned = (1 - total.toDouble / selected.toDouble) * 100
should be:
val percentPruned = (1 - selected.toDouble / total.toDouble) * 100
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
