-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34004/
-----------------------------------------------------------
Review request for drill and Jacques Nadeau.
Bugs: DRILL-1942
https://issues.apache.org/jira/browse/DRILL-1942
Repository: drill-git
Description
-------
Rewritten direct memory allocator. Simplified interface, and use, along with a
means to support additional allocation policies in the future. There are
features in the allocator and in DrillBuf that make finding leaks easier, as
well as better enforcement of limits. New features include transfer of buffers,
and better slicing support.
This is a preliminary patch to get the review started because it touches a lot
of files (readers and record batches were made AutoCloseable in order to cover
cleanup). Subsequent reviews can use the differential view to just see
additional changes. The new allocator is in BaseAllocator.java (along with
derived classes RootAllocator and ChildAllocator); DrillBuf also has
significant changes. Most other changes in other files are just to use newer
interfaces, or to change cleanup() to close(), or to close subordinate objects
that are newly (Auto)Closeable. 1There are still a couple of things to do:
* Some TODO(cwestin)s to clean up tracing and debugging code, as well as adding
javadoc
* Using the AllocatorOwner interface to replace the reallocation mechanism for
FragmentContext and OperatorContext so that the allocator doesn't know anything
about those objects.
Diffs
-----
common/src/main/java/org/apache/drill/common/DrillAutoCloseables.java
PRE-CREATION
common/src/main/java/org/apache/drill/common/DrillCloseables.java
PRE-CREATION
common/src/main/java/org/apache/drill/common/config/DrillConfig.java 522303f
common/src/main/java/org/apache/drill/common/config/NestedConfig.java 3fd885f
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java
9458db2
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveRecordReader.java
5394ee3
contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoRecordReader.java
182f5a4
exec/java-exec/src/main/codegen/templates/AbstractFieldWriter.java 1b5dad1
exec/java-exec/src/main/codegen/templates/BaseWriter.java ada410d
exec/java-exec/src/main/codegen/templates/ComplexWriters.java 49c75d1
exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 1059bfb
exec/java-exec/src/main/codegen/templates/JsonOutputRecordWriter.java ea643f0
exec/java-exec/src/main/codegen/templates/ListWriters.java 6df4248
exec/java-exec/src/main/codegen/templates/MapWriters.java 6ee8035
exec/java-exec/src/main/codegen/templates/NullableValueVectors.java 9d03efb
exec/java-exec/src/main/codegen/templates/ParquetOutputRecordWriter.java
0d24041
exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java c0fba66
exec/java-exec/src/main/codegen/templates/StringOutputRecordWriter.java
f704cca
exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java 7aa7415
exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java 7f80f7a
exec/java-exec/src/main/java/io/netty/buffer/FakeAllocator.java 721aff9
exec/java-exec/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
2ca79f0
exec/java-exec/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java
e332b13
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java be8c7a0
exec/java-exec/src/main/java/org/apache/drill/exec/cache/AbstractStreamSerializable.java
ef488d6
exec/java-exec/src/main/java/org/apache/drill/exec/cache/LoopedAbstractDrillSerializable.java
d2a7458
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java
8e2ce96
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java
5b28f16
exec/java-exec/src/main/java/org/apache/drill/exec/client/DumpCat.java
55d9cf3
exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java
f5a119d
exec/java-exec/src/main/java/org/apache/drill/exec/exception/FragmentSetupException.java
c276846
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MappifyUtility.java
e27234f
exec/java-exec/src/main/java/org/apache/drill/exec/memory/Accountor.java
eb932ad
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationPolicy.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationPolicyAgent.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationReservation.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocatorOwner.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocatorsStatsMXBean.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java
057cfa6
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferAllocator.java
c233ac5
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferLedger.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/ChainedAllocatorOwner.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/ChildAllocator.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/OutOfMemoryException.java
063f1c1
exec/java-exec/src/main/java/org/apache/drill/exec/memory/OutOfMemoryRuntimeException.java
305eabd
exec/java-exec/src/main/java/org/apache/drill/exec/memory/RootAllocator.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/RootAllocatorStatsMXBean.java
PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/TopLevelAllocator.java
a78deb6
exec/java-exec/src/main/java/org/apache/drill/exec/ops/BufferManager.java
c953bb3
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
b108924
exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContext.java
7cc52ba
exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContextImpl.java
6dbd880
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
9e2f210
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java
f56dae3
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScreenCreator.java
5b4d7bd
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java
67062f3
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java
369c0ec
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/TopNBatch.java
9f6bea9
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/WriterRecordBatch.java
15fb7b5
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java
b753574
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
1b90dd8
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java
c1c5cb9
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java
d2282c8
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/FilterRecordBatch.java
5eee9df
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java
dd53477
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java
6466f70
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatchBuilder.java
1187bd6
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/NestedLoopJoinBatch.java
aa4b300
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/limit/LimitRecordBatch.java
eff9e61
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java
ce683cb
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/orderedpartitioner/OrderedPartitionRecordBatch.java
63b7eba
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java
16d1400
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/producer/ProducerConsumerBatch.java
35bf3cd
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortBatch.java
74b7d85
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortRecordBatchBuilder.java
e559ece
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/RemovingRecordBatch.java
aa9297e
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/trace/TraceRecordBatch.java
af45815
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java
66bc3e3
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverBatch.java
66a2092
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java
efd155e
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameRecordBatch.java
86d11d5
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
e88bc67
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/MSortTemplate.java
9b97e1c
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/PriorityQueueCopier.java
161ca6a
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/PriorityQueueCopierTemplate.java
f7786b7
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java
4e348bb
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractSingleRecordBatch.java
dd90cab
exec/java-exec/src/main/java/org/apache/drill/exec/record/RawFragmentBatch.java
edd79ac
exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java
6f10a1c
exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchLoader.java
1b8b7ce
exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorAccessible.java
9db1681
exec/java-exec/src/main/java/org/apache/drill/exec/record/WritableBatch.java
308a8bc
exec/java-exec/src/main/java/org/apache/drill/exec/record/selection/SelectionVector2.java
7a7c012
exec/java-exec/src/main/java/org/apache/drill/exec/record/selection/SelectionVector4.java
69bc78f
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcDecoder.java
74a4afb
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataServer.java
6f8e20b
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryDataBatch.java
914bd00
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserClient.java
a8bad78
exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java
d0a998e
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java
e7a9a3c
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractRecordReader.java
6e27628
exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordReader.java
61ccac5
exec/java-exec/src/main/java/org/apache/drill/exec/store/avro/AvroRecordReader.java
a52fd22
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java
3d789eb
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JsonProcessor.java
4d8d4ba
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/reader/BaseJsonProcessor.java
7833631
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/compliant/CompliantTextRecordReader.java
b2af32d
exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockRecordReader.java
fd97c48
exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockScanBatchCreator.java
74423bf
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ParquetRecordReader.java
58cf321
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet2/DrillParquetReader.java
99ac19c
exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/PojoRecordReader.java
cf98b83
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/MemoryIterator.java
e02b413
exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java
0322f36
exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java
7c77ca2
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseDataValueVector.java
0c6097c
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java
22f0fe7
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java
2fc5bf3
exec/java-exec/src/main/java/org/apache/drill/exec/vector/ValueVector.java
e4a0997
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractContainerVector.java
d14dca6
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractMapVector.java
78846dc
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java
4138839
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java
c061029
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java
e5d48dd
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/DrillBufInputStream.java
1061a5c
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java
095d8c6
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/AbstractBaseWriter.java
ec8c00b
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/ComplexWriterImpl.java
a4a35e2
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/VectorContainerWriter.java
6b6ab46
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/writer/FieldWriter.java
3faa4f7
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/SpoolingRawBatchBuffer.java
2a79e42
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java
49d0c94
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/ForemanException.java
32a99ad
exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java f8ec090
exec/java-exec/src/test/java/org/apache/drill/PlanTestBase.java f909681
exec/java-exec/src/test/java/org/apache/drill/PlanningBase.java da033f7
exec/java-exec/src/test/java/org/apache/drill/QueryTestUtil.java e218d6c
exec/java-exec/src/test/java/org/apache/drill/TestTpchPlanning.java 707ea78
exec/java-exec/src/test/java/org/apache/drill/exec/client/DumpCatTest.java
7c58b19
exec/java-exec/src/test/java/org/apache/drill/exec/compile/bytecode/ReplaceMethodInvoke.java
bc2d929
exec/java-exec/src/test/java/org/apache/drill/exec/expr/ExpressionTest.java
239a099
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestByteComparisonFunctions.java
1e9a47c
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCastFunctions.java
3ba8743
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestMathFunctions.java
4f06a9d
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestNewMathFunctions.java
880184e
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestRepeatedFunction.java
73c7508
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestAllocators.java
74ce225
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestBaseAllocator.java
PRE-CREATION
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestEndianess.java
48ddada
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/SimpleRootExec.java
42d2193
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestCastFunctions.java
ffa8765
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestComparisonFunctions.java
c69c6f5
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestImplicitCastFunctions.java
03c6f41
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestOptiqPlans.java
dc37071
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestSimpleFragmentRun.java
09ba1a5
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestSimpleFunctions.java
d551319
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestStringFunctions.java
d72c1e1
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestUnionExchange.java
9c24f79
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/agg/TestAgg.java
b39566a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/common/TestHashTable.java
b02249d
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/filter/TestSimpleFilter.java
a069078
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoin.java
6c067fe
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestMergeJoin.java
18555c7
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/limit/TestSimpleLimit.java
7cdb41a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/mergereceiver/TestMergingReceiver.java
0122c08
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/project/TestSimpleProjection.java
43c430a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/sort/TestSimpleSort.java
f37624a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/trace/TestTraceMultiRecordBatch.java
b82846e
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/trace/TestTraceOutputDump.java
1cb72ff
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/union/TestSimpleUnion.java
07de27f
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestWriter.java
f4d505d
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestLoad.java
f57e765
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java
1564aea
exec/java-exec/src/test/java/org/apache/drill/exec/server/DrillClientFactory.java
4230518
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestDrillbitResilience.java
da69e9e
exec/java-exec/src/test/java/org/apache/drill/exec/store/TestDirectCodecFactory.java
644144e
exec/java-exec/src/test/java/org/apache/drill/exec/store/ischema/TestInfoSchemaFilterPushDown.java
b6e789b
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/FieldInfo.java
34f60ba
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java
8fdaa72
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java
6326478
exec/java-exec/src/test/java/org/apache/drill/exec/vector/TestSplitAndTransfer.java
4b3aa8a
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/TestEmptyPopulator.java
23cc316
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/fn/TestJsonReaderWithSparseFiles.java
d674d47
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestComplexTypeReader.java
521a41d
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestComplexTypeWriter.java
cb7bef2
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestJsonReader.java
dfa89ca
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestRepeated.java
6e2a2b5
exec/java-exec/src/test/resources/logback.xml 2dcfed9
exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillConnectionImpl.java
74c6655
Diff: https://reviews.apache.org/r/34004/diff/
Testing
-------
Many unit tests run by hand. Currently trying to track down a problem where
some query tests (currently working with TestJsonReader, but several others
exhibit the same behavior) work with assertions enabled, but hang and time out
with assertions disabled.
Will begin submitting for automated testing to see what the state of the rest
of the world is, but wanted to make this available to look at for now because
of its size.
Thanks,
Chris Westin
