Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/761#discussion_r103332364
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/ExternalSortBatch.java
---
@@ -392,22 +448,31 @@ private void configure(DrillConfig config) {
// Set too large and the ratio between memory and input data sizes
becomes
// small. Set too small and disk seek times dominate performance.
- spillBatchSize =
config.getBytes(ExecConstants.EXTERNAL_SORT_SPILL_BATCH_SIZE);
- spillBatchSize = Math.max(spillBatchSize, MIN_SPILL_BATCH_SIZE);
+ preferredSpillBatchSize =
config.getBytes(ExecConstants.EXTERNAL_SORT_SPILL_BATCH_SIZE);
+
+ // In low memory, use no more than 1/4 of memory for each spill batch.
Ensures we
+ // can merge.
+
+ preferredSpillBatchSize = Math.min(preferredSpillBatchSize,
memoryLimit / 4);
--- End diff --
In low memory conditions, restrict the spill batch size to 1/4 of memory.
Why?
* We need to accumulate at least 2 such batches to do a merge. (Now at 1/2
of memory.)
* We need to create an output batch from the two inputs (3/4 of memory).
* Need overhead for other direct memory uses. (Remaining 1/4 of memory.)
Sadly, memory management in Drill is not very precise: batch sizes can't be
predicted with any accuracy. Trying to use, say, 1/3 of memory for the spill
batch would seem more logical. (Two batches into the merge, one out), but the
allocator issues a fatal error if we guess wrong by even one byte. So, we are
forced to be conservative.
If we had better control, or a more forgiving allocator, we could make
different choices.
Also, why try to sort GBs of data in 20 MB? Yet, this is the test case that
had to be solved and that this particular fix enables.
I'm open to suggestions for better solutions; this is a very tricky area...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
