paul-rogers opened a new pull request #2047: DRILL-7675: Work around for partitions sender memory use URL: https://github.com/apache/drill/pull/2047 # [DRILL-7675](https://issues.apache.org/jira/browse/DRILL-7675): Work around for partitions sender memory use ## Description DRILL-7675 describes a combination of factors which exposed a flaw in the partition sender: * The partition sender holds one buffer for each of the receivers, resulting in n^2 buffers total in the system; all on a single machine for a one-node Drill. * Every buffer holds 1024 rows. * The size of each row depends on the row shape. In DRILL-7675, one table has 250+ columns, some nested within repeated maps. Since each needs a vector of 1024 values (or 5 * 1024 or even 5 * 5 * 1024), the total memory size is large. The result is that Drill attempts to allocate many GB of buffers. But, the actual data set is only 2 MB in size. DRILL-7686 describes the needed longer-term redesign. This PR includes a workaround: the ability to reduce the number of rows per send buffer as the receiver count increases. See Documentation below. By enabling the new option, the query will now run in the configuration that the user describes in DRILL-7675. The cost, however, is slower performance, which is exactly what the user was trying to prevent by enabling excessive parallelism. The best workaround in this case (at least with local files) is to go with default parallelism. Also includes a number of cleanup and diagnostic fixes found during the investigation. ## Documentation Adds a new system/session option to allow the buffer size to shrink linearly with the increase in slice count, over some limit: `exec.partition.mem_throttle`: * The default is 0, which leaves the current logic unchanged. * If set to a positive value, then when the slice count exceeds that amount, the buffer size per sender is reduced. * The reduction factor is 1 / (slice count - threshold), with a minimum batch size of 256 records. So, if we set the threshold at 2, and run 10 slices, each slice will get 1024 / 8 = 256 records. This option controls memory, but at obvious cost of increasing overhead. One could argue that this is a good thing. As the number of senders increases, the number of records going to each sender decreases, which increases the time that batches must accumulate before they are sent. If the option is enabled, and buffer size reduction kicks in, you'll find an info-level log message which details the reduction: ``` exec.partition.mem_throttle is set to 2: 10 receivers, reduced send buffer size from 1024 to 256 rows ``` ## Testing Created an ad-hoc test using the query from DRILL-7675. Ran this test with a variety of options, including with the new option enabled and disabled. See DRILL-7675 for a full description of the analysis. Ran the query from DRILL-7675 in the Drill server using the Web console with the new option on and off (along with other variations.) Verified that, with the option off, performance is the same before and after the change. (3 sec on my machine.) Verified that, with the option on, the query completes even with excessive parallelism (though the query does run slower in that case.)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
