[jira] [Created] (SPARK-21034) Filter not getting pushed down the groupBy clause when first() or last() aggregate function is used

[Abhijit Bhole (JIRA)]

Abhijit Bhole created SPARK-21034:
-------------------------------------

             Summary: Filter not getting pushed down the groupBy clause when 
first() or last() aggregate function is used
                 Key: SPARK-21034
                 URL: https://issues.apache.org/jira/browse/SPARK-21034
             Project: Spark
          Issue Type: Bug
          Components: Optimizer
    Affects Versions: 2.1.1
            Reporter: Abhijit Bhole


For example, in my sample code - 

    seriesUserMetricsDF = (userSeriesGameMetricsDF
                       .groupBy(['companyId', "seriesId", 'userId'])
                       .agg(
                            F.last('invitedOn').alias('invitedOn'),
                            F.sum('score').alias('score')))

seriesUserMetricsDF.where(F.col('seriesId') == 12345)

the seriesId filter does not get pushed down to userSeriesGameMetricsDF. In 
Spark 2.1.0 it does.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org