[
https://issues.apache.org/jira/browse/SPARK-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369750#comment-16369750
]
Manan Bakshi commented on SPARK-23463:
--------------------------------------
Hi Marco,
I realized that if you used following code for the same sample:
rdd = sc.textFile("sample").map(lambda x: x.split(", "))
df = rdd.toDF(["dev", "val"])
df = df.filter(df["val"] > 0)
The filter condition actually only filters out values less than 1 even though
it should actually be filtering out values less than 0. This is really weird.
> Filter operation fails to handle blank values and evicts rows that even
> satisfy the filtering condition
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-23463
> URL: https://issues.apache.org/jira/browse/SPARK-23463
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.2.1
> Reporter: Manan Bakshi
> Priority: Critical
> Attachments: sample
>
>
> Filter operations were updated in Spark 2.2.0. Cost Based Optimizer was
> introduced to look at the table stats and decide filter selectivity. However,
> since then, filter has started behaving unexpectedly for blank values. The
> operation would not only drop columns with blank values but also filter out
> rows that actually meet the filter criteria.
> Steps to repro
> Consider a simple dataframe with some blank values as below:
> ||dev||val||
> |ALL|0.01|
> |ALL|0.02|
> |ALL|0.004|
> |ALL| |
> |ALL|2.5|
> |ALL|4.5|
> |ALL|45|
> Running a simple filter operation over val column in this dataframe yields
> unexpected results. For eg. the following query returned an empty dataframe:
> df.filter(df["val"] > 0)
> ||dev||val||
> However, the filter operation works as expected if 0 in filter condition is
> replaced by float 0.0
> df.filter(df["val"] > 0.0)
> ||dev||val||
> |ALL|0.01|
> |ALL|0.02|
> |ALL|0.004|
> |ALL|2.5|
> |ALL|4.5|
> |ALL|45|
>
> Note that this bug only exists in Spark 2.2.0 and later. The previous
> versions filter as expected for both int (0) and float (0.0) values in the
> filter condition.
> Also, if there are no blank values, the filter operation works as expected
> for all versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
