----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25795/#review53925 -----------------------------------------------------------
There is already a facility for restricting the amount of memory used by individual operators. It is set as part of the physical plan. An important case where this is used in External Sort. External Sort is currently the only operator capable of spooling to disk. See DRILL-1329. In the external sort operator, the amount of memory held in the sort is monitored, and when it reaches 95% of the maximum allocation, it spills to disk. This maximum allocation is stored in the popConfig for that Sort. In the case where the new Fragment dynamic limit is less than the operator limit, it would be good for the external sort to also check if it is aobve 95% of this limit as well. - Steven Phillips On Sept. 18, 2014, 9:48 p.m., Parth Chandra wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25795/ > ----------------------------------------------------------- > > (Updated Sept. 18, 2014, 9:48 p.m.) > > > Review request for drill and Steven Phillips. > > > Repository: drill-git > > > Description > ------- > > Provides memory planning at the fragment level. Applies a dynamic limit per > fragment. Allows certain operators to bypass the fragment limit. > > > Diffs > ----- > > exec/java-exec/src/main/java/io/netty/buffer/FakeAllocator.java bc69577 > exec/java-exec/src/main/java/org/apache/drill/exec/memory/Accountor.java > d11f224 > > exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java > 263caa0 > > exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferAllocator.java > 8971eea > > exec/java-exec/src/main/java/org/apache/drill/exec/memory/TopLevelAllocator.java > a8e8a28 > exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java > a888ea7 > exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContext.java > 54edf88 > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/BaseRootExec.java > c2c3144 > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java > 2712e27 > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java > 352deae > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java > d09559d > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java > 29fd80f > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java > 6ff0418 > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverBatch.java > 8e7d9c6 > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java > 52249e9 > > exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java > e8ad311 > > exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java > 5a6ba80 > > exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/SpoolingRawBatchBuffer.java > c7527d5 > > exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java > 083dd95 > > exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestAllocators.java > PRE-CREATION > exec/java-exec/src/test/resources/physical_allocator_test.json PRE-CREATION > > Diff: https://reviews.apache.org/r/25795/diff/ > > > Testing > ------- > > Added new unit test. Ran all unit tests. TPC-H distributed tests are > particularly good tests for this change. > > > Thanks, > > Parth Chandra > >
