[ https://issues.apache.org/jira/browse/DRILL-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886811#comment-15886811 ]
ASF GitHub Bot commented on DRILL-5284: --------------------------------------- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/761#discussion_r103332576 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/ExternalSortBatch.java --- @@ -392,22 +448,31 @@ private void configure(DrillConfig config) { // Set too large and the ratio between memory and input data sizes becomes // small. Set too small and disk seek times dominate performance. - spillBatchSize = config.getBytes(ExecConstants.EXTERNAL_SORT_SPILL_BATCH_SIZE); - spillBatchSize = Math.max(spillBatchSize, MIN_SPILL_BATCH_SIZE); + preferredSpillBatchSize = config.getBytes(ExecConstants.EXTERNAL_SORT_SPILL_BATCH_SIZE); + + // In low memory, use no more than 1/4 of memory for each spill batch. Ensures we + // can merge. + + preferredSpillBatchSize = Math.min(preferredSpillBatchSize, memoryLimit / 4); + + // But, the spill batch should be above some minimum size to prevent complete + // thrashing. + + preferredSpillBatchSize = Math.max(preferredSpillBatchSize, MIN_SPILL_BATCH_SIZE); --- End diff -- Done later when we tally up memory needs and compare it to available memory. We issue an error log message if memory overflow is likely. This way, when a query fails, we can look at the log to see that we knew it would fail due to low memory limits. > Roll-up of final fixes for managed sort > --------------------------------------- > > Key: DRILL-5284 > URL: https://issues.apache.org/jira/browse/DRILL-5284 > Project: Apache Drill > Issue Type: Bug > Affects Versions: 1.10.0 > Reporter: Paul Rogers > Assignee: Paul Rogers > Fix For: 1.10.0 > > > The managed external sort was introduced in DRILL-5080. Since that time, > extensive testing has identified a number of minor fixes and improvements. > Given the long PR cycles, it is not practical to spend a week or two to do a > PR for each fix individually. This ticket represents a roll-up of a > combination of a number of fixes. Small fixes are listed here, larger items > appear as sub-tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)