[jira] [Commented] (SPARK-39833) Filtered parquet data frame count() and show() produce inconsistent results when spark.sql.parquet.filterPushdown is true

2022-08-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575573#comment-17575573 ] Apache Spark commented on SPARK-39833: -- User 'sadikovi' has created a pull request for this issue:

[jira] [Commented] (SPARK-39833) Filtered parquet data frame count() and show() produce inconsistent results when spark.sql.parquet.filterPushdown is true

2022-08-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575571#comment-17575571 ] Apache Spark commented on SPARK-39833: -- User 'sadikovi' has created a pull request for this issue:

[jira] [Commented] (SPARK-39833) Filtered parquet data frame count() and show() produce inconsistent results when spark.sql.parquet.filterPushdown is true

2022-08-04 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575570#comment-17575570 ] Ivan Sadikov commented on SPARK-39833: -- I opened a PR to quickly fix it:

[jira] [Commented] (SPARK-39833) Filtered parquet data frame count() and show() produce inconsistent results when spark.sql.parquet.filterPushdown is true

2022-08-04 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575510#comment-17575510 ] Ivan Sadikov commented on SPARK-39833: -- It appears to be a bug in Parquet-Mr.  There is a

[jira] [Commented] (SPARK-39833) Filtered parquet data frame count() and show() produce inconsistent results when spark.sql.parquet.filterPushdown is true

2022-08-03 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575032#comment-17575032 ] Ivan Sadikov commented on SPARK-39833: -- This is related to case insensitive analysis in Spark. Your

[jira] [Commented] (SPARK-39833) Filtered parquet data frame count() and show() produce inconsistent results when spark.sql.parquet.filterPushdown is true

2022-07-27 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571768#comment-17571768 ] Ivan Sadikov commented on SPARK-39833: -- Interesting, I will take a look. > Filtered parquet data

[jira] [Commented] (SPARK-39833) Filtered parquet data frame count() and show() produce inconsistent results when spark.sql.parquet.filterPushdown is true

2022-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571763#comment-17571763 ] Hyukjin Kwon commented on SPARK-39833: -- Seems like a bug from Parquet side in rowgroup filtering.