GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/8496
[SPARK-10065][SQL] avoid the extra copy when generate unsafe array
The reason for this extra copy is that we iterate the array elements twice:
one for calculate size, one for write to `UnsafeArray`.
A simple solution is using the same skill in generating unsafe row: grow
the byte[] buffer when needed. Then we don't need to know the data size ahead
and only need to iterate once.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark avoid-copy
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8496.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8496
----
commit 56f226842f9668ccfde947fafd07a4400dc38a5a
Author: Wenchen Fan <cloud0...@outlook.com>
Date: 2015-08-28T05:44:52Z
make sure input.primitive is always variable name not code
commit 79cfc445e3e4ae87ab5d4e907c27d57267c37626
Author: Wenchen Fan <cloud0...@outlook.com>
Date: 2015-08-28T06:28:25Z
avoid the extra copy when generate unsafe array
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
