[
https://issues.apache.org/jira/browse/DRILL-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886816#comment-15886816
]
ASF GitHub Bot commented on DRILL-5284:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/761#discussion_r103332807
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/ExternalSortBatch.java
---
@@ -765,12 +838,12 @@ private void processBatch() {
spillFromMemory();
}
- // Sanity check. We should now be above the spill point.
+ // Sanity check. We should now be below the buffer memory maximum.
long startMem = allocator.getAllocatedMemory();
- if (memoryLimit - startMem < spillPoint) {
- logger.error( "ERROR: Failed to spill below the spill point. Spill
point = {}, free memory = {}",
- spillPoint, memoryLimit - startMem);
+ if (startMem > bufferMemoryPool) {
+ logger.error( "ERROR: Failed to spill above buffer limit. Buffer
pool = {}, memory = {}",
+ bufferMemoryPool, startMem);
--- End diff --
We could. But, at this point, it is a potential problem, not a real one.
Maybe the input has no rows; we won't overflow memory. Maybe just one or two
rows and we'll be fine.
This warning says, "if we continue as we are now, and we have large amounts
of data, we'll run off the rails." It helps explain any later OOM error that
occurs.
> Roll-up of final fixes for managed sort
> ---------------------------------------
>
> Key: DRILL-5284
> URL: https://issues.apache.org/jira/browse/DRILL-5284
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.10.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.10.0
>
>
> The managed external sort was introduced in DRILL-5080. Since that time,
> extensive testing has identified a number of minor fixes and improvements.
> Given the long PR cycles, it is not practical to spend a week or two to do a
> PR for each fix individually. This ticket represents a roll-up of a
> combination of a number of fixes. Small fixes are listed here, larger items
> appear as sub-tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
