GitHub user cloud-fan reopened a pull request:
https://github.com/apache/spark/pull/15082
[SPARK-17528][SQL] MutableProjection should not cache content from the
input row
## What changes were proposed in this pull request?
For performance reasons, `UnsafeRow.getString`, `getStruct`, etc. return a
"pointer" that points to a memory region of this unsafe row. This makes the
unsafe projection a little dangerous, because all of its output rows share one
instance.
When we implement SQL operators, we should be careful to not cache the
input rows because they may be produced by unsafe projection from child
operator and thus its content may change overtime.
However, `GenerateMutableProjection` breaks this and may cache the content
in input rows. The sort based aggregate use it, but this bug is not exposed
because sort based aggregate always do an extra projection for the input row.
This PR fixes the bug of `GenerateMutableProjection` and some related bugs
in complex data copy, and remove the useless projection in sort based aggregate.
## How was this patch tested?
some new tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark agg-bug
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15082.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15082
----
commit 75b57490d1be2ca4d7ca88bcd4d6d9461d470ce8
Author: Wenchen Fan <wenc...@databricks.com>
Date: 2016-09-13T17:40:16Z
MutableProjection should not cache content from the input row
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
