Burak Yavuz created SPARK-21153:
-----------------------------------
Summary: Time windowing for tumbling windows can use a project
instead of expand + filter
Key: SPARK-21153
URL: https://issues.apache.org/jira/browse/SPARK-21153
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.1.1
Reporter: Burak Yavuz
Assignee: Burak Yavuz
Time windowing in Spark currently performs an Expand + Filter, because there is
no way to guarantee the amount of windows a timestamp will fall in, in the
general case. However, for tumbling windows, a record is guaranteed to fall
into a single bucket. In this case, doubling the number of records with Expand
is wasteful, and can be improved by using a simple Projection instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
