[
https://issues.apache.org/jira/browse/DRILL-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481337#comment-16481337
]
Paul Rogers commented on DRILL-6429:
------------------------------------
Bit of the back-story here. Originally, there was no `setLenient()`. We added
it just for sort.
Drill's allocator model enforces strict per-operator limits. Until a year ago,
operators made no real attempt to honor the limit; so the limit was the only
tool available to prevent run-away operator usage. Of course, the limit was of
magical value because, for most operators, it was set at 10 GB...
For sort, we did our best to modify the code to honor those limits for
spilling, measuring input batch size and so on. As a result, the operator
itself enforces its own limit.
Now, it is nearly impossible to get everything right since the Sort has no
control over its input batch size. The result is that when the moon is full and
the starts are aligned just right, the Sort will exceed the hard-coded limit by
some few percent.
Should we punish the user by killing their query? We decided not to. So, we
introduced the lenient mode.
The thought is, since most operators made no attempt at all to limit memory, it
makes no sense to punish the user because Sort happened to be off by a few
percent.
Given all this, what is the reason to change our thinking now? Can the sort do
a better job of predicting memory usage? Does sort now have control over its
input batch size?
> SortImpl Should Not Use BufferAllocator.setLenient()
> ----------------------------------------------------
>
> Key: DRILL-6429
> URL: https://issues.apache.org/jira/browse/DRILL-6429
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Timothy Farkas
> Assignee: Timothy Farkas
> Priority: Major
>
> The Sort operator should always obey its memory limit. This is required for
> resource management.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
