GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/20890
[WIP][SPARK-23779][SQL] TaskMemoryManager and UnsafeSorter related classes
use MemoryBlock
## What changes were proposed in this pull request?
Waiting for merging #19222
This PR tries to use `MemoryBlock` in `TaskMemoryManager` and
`UnsafeSorter` related classes. There are two advantages to use `MemoryBlock`.
1. Has clean API calls rather than using a Java array or `PlatformMemory`
2. Improve runtime performance of memory access instead of using `Object`
with `Platform.get/put...`.
## How was this patch tested?
Used existing UTs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark SPARK-23779
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20890.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20890
----
commit f427ca2a0f30d70f42b4755d0d72318cfa0e4c77
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-13T10:16:19Z
introduce ByteArrayMemoryBlock, IntArrayMemoryBlock, LongArrayMemoryBlock,
and OffheaMemoryBlock
commit 5d7ccdb0e845afcaa430bac3d21b519d35d1e6f4
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-13T17:15:25Z
OffHeapColumnVector uses UnsafeMemoryAllocator
commit 251fa09d4421085a6c97f56bb38108980a79b5c8
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-13T17:27:09Z
UTF8String uses UnsafeMemoryAllocator
commit 790bbe7f3ac52b2b1e4375684998737d6127a552
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-13T17:36:57Z
Platform.copymemory() in UsafeInMemorySorter uses new MemoryBlock
commit 93a792e7729f213fcef66f0ed9b33f45259d1ec3
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-14T15:34:48Z
address review comments
commit 0beab0308cdac57e85daf7794170a0ca899ab568
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-14T17:53:33Z
fix test failures (e.g. String in UnsafeArrayData)
commit fcf764c1aebdc847675f710c17ec8477d6022a40
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-18T16:13:13Z
fix failures
commit d2d2e50f8a2baf41d5b85127bf888da6f8bca343
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-21T18:45:55Z
minor update of UTF8String constructor
commit f5e10bb52c33856ddd3e1b1f8483b170e0167c53
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-22T11:00:12Z
rename method name
commit 1905e8ca4b3b8200fa56f5fc91899bb420a07628
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-22T11:01:13Z
remove unused code
commit 7778e586e94130749cec3f54a60b6fb24514647a
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-22T11:02:30Z
update arrayEquals
commit 4f96c82b151b78641bcfc92a65913048b055cfee
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-22T13:18:01Z
rebase master
commit d1d6ae90589c0fae6c64af0cb95696e270446228
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-22T14:51:40Z
make more methods final
commit 914dcd11d0d5ef014284868f2794cb4e5baa0958
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-22T15:39:57Z
make fill method final in MemoryBlock
commit 336e4b7bfd7edcb861edeac3ca115dead785b68a
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-23T11:35:15Z
fix test failures
commit 5be9ccb163832e1895b045730a06e79eb3b171cf
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-24T14:00:40Z
add testsuite
commit 43e6b572bd893bd42e58df930a34b7e31549a49a
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-24T18:10:49Z
pass concrete type to the first argument of Platform.get*/put* to get
better performance
commit 05f024e566f828e9c3f836430c9c7b34da5e954b
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-28T01:51:38Z
rename methods related to hash
commit 9071cf6449123400f3d774664e1709337b05c555
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-28T01:52:48Z
added methods for MemoryBlock
commit 37ee9fa07a8f8faaeb097164422bb15958fa4b1c
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-28T03:44:30Z
rebase with master
commit d0b5d59bb31fe2845477ee243008992686e2f2a2
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-09-28T04:37:25Z
fix scala style error
commit 5cdad44717ccb510d4114d14cfff304bef9f5bb4
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-10-14T07:29:14Z
use MemoryBlock in Murmur3 for performance reason
commit 91028fa2ae34bb3ae667692112b8455d4394cbbd
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-10-14T07:29:30Z
fix typo in comment
commit 0210bd1e5f46f81617a35493d2cd0b737b4cf85d
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-10-29T12:32:38Z
address review comment
commit df6dad3762f4e918d503df75ae8fce052af8bf43
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2017-11-28T06:08:52Z
rebase with master
commit 1fa47a8c291ab54a8c1e386737769f467a76a672
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2018-02-20T12:17:26Z
fix failures
commit 01f9c8e8146ff8d21d18feba59c0c2ad83299e2a
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2018-02-20T17:30:11Z
fix failures in ArrowColumnVectorSuite and FeatureHasherSuite
commit 2ed8f82f13288e00e6352bdccbac0dd890c91715
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2018-02-21T17:48:41Z
address review comments
rename LongArrayMemoryBlock to OnHeapMemoryBlock
remove intArrayMemoryBlock
separate copyMemory to copyFrom and writeTo and make them non-static
add assertion for range check
commit 5e3afd11a2dc76d2cd23264b4052abbf3f5d7e9d
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2018-02-23T16:20:21Z
address review comments
commit 95fbdee04e9137938cdc76f7f4573116720357f5
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2018-02-24T00:22:44Z
fix test failures
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
