[
https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588101#comment-16588101
]
Jungtaek Lim commented on SPARK-24763:
--------------------------------------
[~tdas]
One question regarding fix version: I guess we still don't cut the Spark 2.4
branch, so I expect everything on master will be included to Spark 2.4. Do I
understand correctly? Because we have 4 issues marked fix version to only
3.0.0, so curious which version they're being included.
https://issues.apache.org/jira/browse/SPARK-24763?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%203.0.0
> Remove redundant key data from value in streaming aggregation
> -------------------------------------------------------------
>
> Key: SPARK-24763
> URL: https://issues.apache.org/jira/browse/SPARK-24763
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Major
> Fix For: 2.4.0, 3.0.0
>
>
> Key/Value of state in streaming aggregation is formatted as below:
> * key: UnsafeRow containing group-by fields
> * value: UnsafeRow containing key fields and another fields for aggregation
> results
> which data for key is stored to both key and value.
> This is to avoid doing projection row to value while storing, and joining key
> and value to restore origin row to boost performance, but while doing a
> simple benchmark test, I found it not much helpful compared to "project and
> join". (will paste test result in comment)
> So I would propose a new option: remove redundant in stateful aggregation.
> I'm avoiding to modify default behavior of stateful aggregation, because
> state value will not be compatible between current and option enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
