GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/20890
[WIP][SPARK-23779][SQL] TaskMemoryManager and UnsafeSorter related classes use MemoryBlock ## What changes were proposed in this pull request? Waiting for merging #19222 This PR tries to use `MemoryBlock` in `TaskMemoryManager` and `UnsafeSorter` related classes. There are two advantages to use `MemoryBlock`. 1. Has clean API calls rather than using a Java array or `PlatformMemory` 2. Improve runtime performance of memory access instead of using `Object` with `Platform.get/put...`. ## How was this patch tested? Used existing UTs You can merge this pull request into a Git repository by running: $ git pull https://github.com/kiszk/spark SPARK-23779 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20890.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20890 ---- commit f427ca2a0f30d70f42b4755d0d72318cfa0e4c77 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-13T10:16:19Z introduce ByteArrayMemoryBlock, IntArrayMemoryBlock, LongArrayMemoryBlock, and OffheaMemoryBlock commit 5d7ccdb0e845afcaa430bac3d21b519d35d1e6f4 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-13T17:15:25Z OffHeapColumnVector uses UnsafeMemoryAllocator commit 251fa09d4421085a6c97f56bb38108980a79b5c8 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-13T17:27:09Z UTF8String uses UnsafeMemoryAllocator commit 790bbe7f3ac52b2b1e4375684998737d6127a552 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-13T17:36:57Z Platform.copymemory() in UsafeInMemorySorter uses new MemoryBlock commit 93a792e7729f213fcef66f0ed9b33f45259d1ec3 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-14T15:34:48Z address review comments commit 0beab0308cdac57e85daf7794170a0ca899ab568 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-14T17:53:33Z fix test failures (e.g. String in UnsafeArrayData) commit fcf764c1aebdc847675f710c17ec8477d6022a40 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-18T16:13:13Z fix failures commit d2d2e50f8a2baf41d5b85127bf888da6f8bca343 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-21T18:45:55Z minor update of UTF8String constructor commit f5e10bb52c33856ddd3e1b1f8483b170e0167c53 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-22T11:00:12Z rename method name commit 1905e8ca4b3b8200fa56f5fc91899bb420a07628 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-22T11:01:13Z remove unused code commit 7778e586e94130749cec3f54a60b6fb24514647a Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-22T11:02:30Z update arrayEquals commit 4f96c82b151b78641bcfc92a65913048b055cfee Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-22T13:18:01Z rebase master commit d1d6ae90589c0fae6c64af0cb95696e270446228 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-22T14:51:40Z make more methods final commit 914dcd11d0d5ef014284868f2794cb4e5baa0958 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-22T15:39:57Z make fill method final in MemoryBlock commit 336e4b7bfd7edcb861edeac3ca115dead785b68a Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-23T11:35:15Z fix test failures commit 5be9ccb163832e1895b045730a06e79eb3b171cf Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-24T14:00:40Z add testsuite commit 43e6b572bd893bd42e58df930a34b7e31549a49a Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-24T18:10:49Z pass concrete type to the first argument of Platform.get*/put* to get better performance commit 05f024e566f828e9c3f836430c9c7b34da5e954b Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-28T01:51:38Z rename methods related to hash commit 9071cf6449123400f3d774664e1709337b05c555 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-28T01:52:48Z added methods for MemoryBlock commit 37ee9fa07a8f8faaeb097164422bb15958fa4b1c Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-28T03:44:30Z rebase with master commit d0b5d59bb31fe2845477ee243008992686e2f2a2 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-09-28T04:37:25Z fix scala style error commit 5cdad44717ccb510d4114d14cfff304bef9f5bb4 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-10-14T07:29:14Z use MemoryBlock in Murmur3 for performance reason commit 91028fa2ae34bb3ae667692112b8455d4394cbbd Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-10-14T07:29:30Z fix typo in comment commit 0210bd1e5f46f81617a35493d2cd0b737b4cf85d Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-10-29T12:32:38Z address review comment commit df6dad3762f4e918d503df75ae8fce052af8bf43 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2017-11-28T06:08:52Z rebase with master commit 1fa47a8c291ab54a8c1e386737769f467a76a672 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2018-02-20T12:17:26Z fix failures commit 01f9c8e8146ff8d21d18feba59c0c2ad83299e2a Author: Kazuaki Ishizaki <ishizaki@...> Date: 2018-02-20T17:30:11Z fix failures in ArrowColumnVectorSuite and FeatureHasherSuite commit 2ed8f82f13288e00e6352bdccbac0dd890c91715 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2018-02-21T17:48:41Z address review comments rename LongArrayMemoryBlock to OnHeapMemoryBlock remove intArrayMemoryBlock separate copyMemory to copyFrom and writeTo and make them non-static add assertion for range check commit 5e3afd11a2dc76d2cd23264b4052abbf3f5d7e9d Author: Kazuaki Ishizaki <ishizaki@...> Date: 2018-02-23T16:20:21Z address review comments commit 95fbdee04e9137938cdc76f7f4573116720357f5 Author: Kazuaki Ishizaki <ishizaki@...> Date: 2018-02-24T00:22:44Z fix test failures ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org