GitHub user hyunsik opened a pull request:
https://github.com/apache/tajo/pull/133
TAJO-907: Implement off-heap tuple block and zero-copy tuple.
Hi folks,
One week ago, I contributed run-time code generation for computation
efficiency and reduction of objection creations. This work has also the similar
purpose.
This patch includes off-heap memory and row block (a list of tuple) using
off-heap memory container. Also, it provides various utility classes for them.
In detail, this patch includes lots of changes.
* OffHeapMemory class
* OffHeapRowBlock class
* ZeroCopyTuple, which just points to actual row record stored in
OffHeapRowBlock.
* RowWriter interface
* RowBlockReader interface and OffHeapBlockReader
* TupleBuilder and BaseTupleBuilder
* HeapTuple, which keeps fields in byte array instead of Datum array.
* HeapTuple internally UnSafe to read and store field values in byte
array
* TupleComparatorCompiler, which is run-time code compiler for
TupleComparator
* It reduces branches even though tuple comparator actually should
consider complex logic with all null-first/last, ascending, and descending
orders.
* Others
Later, I'll replace current VTuple by HeapTuple or ZeroCopyTuple. Also, I'm
planning the improvement to replace current pull-iterator model of execution
engine by the push-based block iterator model. I'll describe it later in
another jira issue.
Thanks,
Hyunsik
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hyunsik/tajo TAJO-907
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/133.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #133
----
commit ded746bc691fa68b06019909c740a9fe945ab0a5
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-23T17:38:27Z
initial work for direct memory tuple.
commit c9414c012fdd388fb51e0a5bd81d2dca31930188
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-24T15:43:34Z
Implemented DirectRowBlock and its unit tests.
commit 943a29df75f9d583455002be4f65696052950b4d
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-24T16:11:52Z
Added benchmark test.
commit b6b3b4ffe39d4b2e5f75fb6749e8868751f51f17
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-24T16:31:22Z
Improved interfaces and refactored class hierarchy.
commit b424269b3c6e4728bea1c1e5248a30af611a6ead
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-24T23:29:58Z
added sort benchmark and add get() method to UnSafeTuple.
commit ef07dc9a93e66650f532e5fdc7753918f7bbf14f
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-25T04:23:59Z
Add timestamp data type support to RowOrientedRowBlock.
commit cb94b74a7f79cb074669f09252f8069fd8933db8
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-25T05:50:18Z
Cleanup unit tests and benchmarks.
commit 8d02ba7a1e2d0f4a8edb1cbc7910abc11e6e386d
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-25T06:29:35Z
Removed duplicated method cleanByteBuffer() from StorageUtil.
commit d783d65e9db97883f793ad24d7daabd36a514e3f
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-25T07:43:09Z
Refactored UnSafeTuple.
commit ab241625982296ed11813f0e9d95147fa4dddf21
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-25T11:25:33Z
Refactored SchemaUtil to tajo-catalog, improved DirectRawFileWriter.
commit 75949fa1455e75347b936bcce31c61a726581942
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-25T16:33:37Z
Implemented zero copy and zero (de)serialization file format.
commit 43d45be517e3a9b3c4d9c363a2e7e05e0def0acd
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-25T17:01:36Z
Implemented legacy addTuple method.
commit 6ed1df9818a79ac71f78bc9beb1083abbc50f02b
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-26T01:20:48Z
Implemented VTuple to UnSafeTuple encoder.
commit 72179ef4083c66ebe251482dd50913ef0ca19107
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-26T02:14:11Z
Improved RowOrientedRowBlock to take exiting Tuple.
commit d78d207455a697ca47c44ed6e4200c6b0b9ad07f
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-26T06:47:59Z
Fixed many bugs of DirectRawFileScanner.
commit 6b8f0d4484f3d546e7cff6dddc28dcc117171b59
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-26T07:43:07Z
External sort works well, but random access causes performance degradation.
commit afda01e3cae8ac88938d1a504c9faca0aca29845
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-27T14:49:08Z
Fixed many bugs and refactored tests.
commit a0859fd47b7e9d8c15e30a80c2d27e7d81eccfb3
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-28T04:07:33Z
Fixed reset bugs and resource leak in PairWiseMerger.
commit df91243afa6710de66a0386291969252bd0702f5
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-29T07:31:30Z
Implemented the basic of tuple comparator generation.
commit 31e418b2fbf2df2f1e192d34df22139f8c0a684f
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-29T14:46:09Z
Implemented TupleComparatorCompiler and UnSafeTupleTextComparator.
commit 738dd06421cd159c6c24390f29da94bb0a400602
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-29T15:07:16Z
Improving Null last and Null first of TupleComparatorCompiler.
commit 2930bd98d6011bcd45d3a2032c672e28594f3515
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-29T15:13:11Z
Renamed UnSafeComparator.
commit 06675b7cb1934563585127c62c5457459254a49b
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-29T15:19:10Z
Fixed a bug of UnSafeTupleTextComparator.
commit 1f0fa60929c2ae9858eb3f0bd116a9f2855f287e
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-29T16:33:46Z
Improved null first and last of comparator.
commit 8ab577460c2d6fa4983014963f574611014f9d51
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-31T06:50:07Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into
DirectMemTuple
commit 631dd21b54f9994736a7d7ca97f67717483567fe
Author: Hyunsik Choi <[email protected]>
Date: 2014-08-31T12:53:51Z
Fixed offset mismatch bug.
commit e4e6fa3b7a8c7e818294e8fcaf20baba5f9cd9b5
Author: Hyunsik Choi <[email protected]>
Date: 2014-09-01T01:22:23Z
Passed all unit tests.
commit fcc59d38cb99afd97a2bec03f4c39344e416df00
Author: Hyunsik Choi <[email protected]>
Date: 2014-09-01T06:45:48Z
Improved TupleComparatorCompiler.
commit 6c9084a1f8e88fb22ed226bc81884bcf452b5439
Author: Hyunsik Choi <[email protected]>
Date: 2014-09-01T06:55:08Z
renamed UnSafeComparer.
commit 24200be8d2165eeb8c505d9fe3df305db3cce5b3
Author: Hyunsik Choi <[email protected]>
Date: 2014-09-01T12:01:14Z
Fixed isNull and many bugs related to ordering.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
