jnturton commented on issue #2789: URL: https://github.com/apache/drill/issues/2789#issuecomment-1500895198
Okay. 1. After doing some tests I think there may be a bug affecting the direct memory counter in the most recent version of Drill. 2. If you hover over the direct memory usage info icon you'll see that, unlike for the heap, Drill reports current usage as a percentage of _peak_ usage, not of the configured capacity. 3. Your query is probably getting executed entirely, or almost, on the RDBMS. Drill tries to "push down" query execution to the source system when it can. You can prevent this by running two queries, first a CTAS that fetches data from DEF_TABLE without performing any operations and writes it somewhere, perhaps the temp workspace. Secondly, do all of the rest of the logic in a SELECT based on the table that was created by the CTAS. Note however that this will require 180m rows to be transferred from the RDBMS to Drill over JDBC which will itself need a lot of time (but may not always need to be repeated). I'll report back here about the direct memory counter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
