jnturton commented on PR #2610: URL: https://github.com/apache/drill/pull/2610#issuecomment-1193118355
@luocooong the main goal of this PR is to solve issues in RC0 of Drill 1.20.2 and produce RC1. Unrelated to that, I can expand a bit more on my reasoning for the commit disabling the heap memory usage limiter in the REST server by default. Since we stream rather than buffer REST API query results now, we no longer have an excuse for runaway heap usage. Further, while preventing the next REST query from running can save the Drillbit from an OOM death, when the user complains that they cannot run their query an admin is very likely to restart that Drillbit _anyway_. So we probably only delayed the inevitable even though the landing was made a bit softer and other processing on the Drillbit may have been able to continue. Since now the only things that can cause runaway heap usage (that I can see) under a constant load are leaks, unecessary object references in Java's case, I think we should bite the bullet and fail hard with OOM when we exhaust the heap. The Drillbit can easily be reincarnated upon its death by systemd or another supervisor, which is preferable to having it limp on declining queries arriving over REST. We also get to stop applying our own heap management rule which may interact badly with the GC in use. E.g. if we start turning REST queries away at 85% heap usage but 90% heap usage would have stirred the user's GC to do a deep clean (in reality I think they all kick in considerably earlier but I'm really making a design argument here) then we are working against our own JVM. Lastly any ensuing bug reports overtly containing "OutOfMemoryError" are more likely, I believe, to send us hunting heap memory leaks. cc @paul-rogers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
