Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/928#discussion_r143088158
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -392,10 +419,12 @@ drill.exec.options: {
exec.query_profile.debug_mode: false,
exec.query_profile.save: true,
exec.queue.enable: false,
- exec.queue.large: 10,
- exec.queue.small: 100,
+ exec.queue.large: 2,
--- End diff --
As for the smaller values... Drill ships with a default of 8 GB heap, and
allows 2 GB per query per node by default. Drill only has enough memory to run
3 "normal-sized" queries. These new numbers are, in fact, wildly optimistic
that, in that configuration, we can run 10 small and two large queries. We
assume that each will get (8 GB * .8) / 30 = ~210 MB per small query, about 2
GB per large query.
In fact, in explaining this, I realized that the small queue number is
still too large for the default install and have reduced the numbers to large =
2, small = 4. Memory for small is now (8 GB * 0.8) / 15 = ~430 MB, with large
now ~ 2 GB. This maps a "large" query to the default memory-per-query-per-node.
Granted, the 20% overhead is a SWAG, we have evidence that it is sometimes
larger. Fixing that is another project...
---
