> On Sept. 19, 2014, 2:45 a.m., Steven Phillips wrote: > > There is already a facility for restricting the amount of memory used by > > individual operators. It is set as part of the physical plan. An important > > case where this is used in External Sort. External Sort is currently the > > only operator capable of spooling to disk. See DRILL-1329. In the external > > sort operator, the amount of memory held in the sort is monitored, and when > > it reaches 95% of the maximum allocation, it spills to disk. This maximum > > allocation is stored in the popConfig for that Sort. In the case where the > > new Fragment dynamic limit is less than the operator limit, it would be > > good for the external sort to also check if it is aobve 95% of this limit > > as well.
I can add a simple check in the external sort to check if the memory use has reached 0.95 of the fragment limit. Is there a unit test for external sort I can use to test the changes? Longer term, the data in popConfig should be populated for all operators not just for external sort and the fragment limit should be set as a function of the values in popConfig as well as the available memory. Though that really needs more discussion. - Parth ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25795/#review53925 ----------------------------------------------------------- On Sept. 18, 2014, 9:48 p.m., Parth Chandra wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25795/ > ----------------------------------------------------------- > > (Updated Sept. 18, 2014, 9:48 p.m.) > > > Review request for drill and Steven Phillips. > > > Repository: drill-git > > > Description > ------- > > Provides memory planning at the fragment level. Applies a dynamic limit per > fragment. Allows certain operators to bypass the fragment limit. > > > Diffs > ----- > > exec/java-exec/src/main/java/io/netty/buffer/FakeAllocator.java bc69577 > exec/java-exec/src/main/java/org/apache/drill/exec/memory/Accountor.java > d11f224 > > exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java > 263caa0 > > exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferAllocator.java > 8971eea > > exec/java-exec/src/main/java/org/apache/drill/exec/memory/TopLevelAllocator.java > a8e8a28 > exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java > a888ea7 > exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContext.java > 54edf88 > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/BaseRootExec.java > c2c3144 > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java > 2712e27 > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java > 352deae > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java > d09559d > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java > 29fd80f > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java > 6ff0418 > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverBatch.java > 8e7d9c6 > > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java > 52249e9 > > exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java > e8ad311 > > exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java > 5a6ba80 > > exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/SpoolingRawBatchBuffer.java > c7527d5 > > exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java > 083dd95 > > exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestAllocators.java > PRE-CREATION > exec/java-exec/src/test/resources/physical_allocator_test.json PRE-CREATION > > Diff: https://reviews.apache.org/r/25795/diff/ > > > Testing > ------- > > Added new unit test. Ran all unit tests. TPC-H distributed tests are > particularly good tests for this change. > > > Thanks, > > Parth Chandra > >
