GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/19472
Use MemoryBlock in UnsafeRow, UnsafeArrayData, and UnsafeMapData
## What changes were proposed in this pull request?
Waiting for merging #19222
This PR tries to use `MemoryBlock` in `UnsafeRow`, `UnsafeArray`, and
`UnsafeMap`. There are two advantages to use `MemoryBlock`
1. Increase flexibility of choosing memory type (e.g `byte[]`, `int[]`,
`long[]`, or `DirectBuffer (offheap)`).
2. Improve runtime performance of memory access instead of using `Object`.
## How was this patch tested?
Use existing testsuites
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark SPARK-22246
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19472.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19472
----
commit 28e335d18da8b9c17b18d316943a90bfb290ee45
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-13T10:16:19Z
introduce ByteArrayMemoryBlock, IntArrayMemoryBlock, LongArrayMemoryBlock,
and OffheaMemoryBlock
commit 47b05f3d5a1c2fc2f692cf42745eef9e738eb540
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-13T17:15:25Z
OffHeapColumnVector uses UnsafeMemoryAllocator
commit ae891eee8e214b63d204e794e2cb9f43f8a71e10
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-13T17:27:09Z
UTF8String uses UnsafeMemoryAllocator
commit 1f97cfe6db3e404c090df7ae9a8e1b5b88ca254b
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-13T17:36:57Z
Platform.copymemory() in UsafeInMemorySorter uses new MemoryBlock
commit 5472ab84daf37579ad9f6ca6d17beb652ddd5cfd
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-14T15:34:48Z
address review comments
commit fa3aa7eb3b87bfddf3923fe4bed1d4fa1081ae2e
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-14T17:53:33Z
fix test failures (e.g. String in UnsafeArrayData)
commit 186aa822459f314605e6b517da7148c1507b6401
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-18T16:13:13Z
fix failures
commit 15eab8fff1316cc0e1857e6cb0fdedfba2dc2d53
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-21T18:45:55Z
minor update of UTF8String constructor
commit dcc45804504c244c1dcd8898aa2004bdc9e0245f
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-22T11:00:12Z
rename method name
commit e43587efabbb6511ef1e2df3c86d50ac7ee75c4e
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-22T11:01:13Z
remove unused code
commit b4a9de1d7edf665e4092b68b55a72d0ad6355fd3
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-22T11:01:33Z
reformat
commit 0ea3cff303f5cd1d3bd9290ca3290fad977f5924
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-22T11:02:30Z
update arrayEquals
commit 92d9b3b91f654d9cbc7923132afe6118d90781e8
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-22T13:18:01Z
rebase master
commit 791be2e681d934f9d64faf984949b65228804d11
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-22T14:51:40Z
make more methods final
commit d1ffe0a67c6dc7888114c49cdeae61f125cf55df
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-22T15:39:57Z
make fill method final in MemoryBlock
commit 614933e042b9266118383daf254478bd3a9bf438
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-23T11:35:15Z
fix test failures
commit 81b454df13bbc48b7ae7638bb80b8fe7de820bbf
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-24T14:00:40Z
add testsuite
commit 81a0964de95ab38d3d06d8226caae37de0243d21
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-24T18:10:49Z
pass concrete type to the first argument of Platform.get*/put* to get
better performance
commit c5f71f29e24306819824714e562aba2353777392
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-28T01:51:38Z
rename methods related to hash
commit a82220adc4b1f1ccc109c1922f889b2301d55a13
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-28T01:52:48Z
added methods for MemoryBlock
commit 5ad4c542f9371d12a8391139cee45388c5175764
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-28T03:44:30Z
rebase with master
commit 67d5a8b48cc851573bad4b825e3612a85cdbe432
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-09-28T04:37:25Z
fix scala style error
commit 03bc8f34df17de0d0cd927c2ea50673b60fca4af
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-10-11T02:14:34Z
rebase with master
commit e51fb6ae246a660ec2b0bdd1f6a53a0987d4a063
Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Date: 2017-10-11T10:10:21Z
Use MemoryBlock in UnsafeRow, UnsafeArrayData, and UnsafeMapData
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
