[
https://issues.apache.org/jira/browse/DRILL-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16508493#comment-16508493
]
ASF GitHub Bot commented on DRILL-6477:
---------------------------------------
kkhatua commented on a change in pull request #1309: DRILL-6477: Drillbit
crashes with OOME (Heap) for a large WebUI query
URL: https://github.com/apache/drill/pull/1309#discussion_r194502221
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java
##########
@@ -68,8 +78,37 @@ public QueryResult run(final WorkManager workManager, final
WebUserConnection we
// Submit user query to Drillbit work queue.
final QueryId queryId =
workManager.getUserWorker().submitWork(webUserConnection, runQuery);
+ heapMemoryFailureThreshold =
workManager.getContext().getConfig().getDouble(
ExecConstants.HTTP_QUERY_FAIL_LOW_HEAP_THRESHOLD );
+ boolean isComplete = false;
+ boolean nearlyOutOfHeapSpace = false;
+ float usagePercent = getHeapUsage();
+
// Wait until the query execution is complete or there is error submitting
the query
- webUserConnection.await();
+ logger.debug("Wait until the query execution is complete or there is error
submitting the query");
+ do {
+ try {
+ isComplete = webUserConnection.await(TimeUnit.SECONDS.toMillis(1));
/*periodically timeout to check heap*/
+ } catch (Exception e) { }
+
+ usagePercent = getHeapUsage();
+ if (usagePercent > heapMemoryFailureThreshold) {
+ nearlyOutOfHeapSpace = true;
+ }
+ } while (!isComplete && !nearlyOutOfHeapSpace);
+
+ //Fail if nearly out of heap space
+ if (nearlyOutOfHeapSpace) {
+ workManager.getBee().getForemanForQueryId(queryId)
+ .addToEventQueue(QueryState.FAILED,
+ UserException.resourceError(
+ new Throwable(
+ "Query submitted through the Web interface was failed due
to diminishing free heap memory ("+ Math.floor(((1-usagePercent)*100)) +"%
free). "
Review comment:
I thought it would be useful to show the level at which the free memory was
prior to cancellation. I suspect that people will see a GC'ed Drillbit after
the cancellation and wonder why Drill is complaining of no sufficient heap.
Hence, the wording.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Drillbit hangs/crashes with OOME Java Heap Space for a large query through
> WebUI
> --------------------------------------------------------------------------------
>
> Key: DRILL-6477
> URL: https://issues.apache.org/jira/browse/DRILL-6477
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
> Affects Versions: 1.13.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Priority: Major
> Fix For: 1.14.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For queries submitted through the WebUI and retrieving a large resultset, the
> Drillbit often hangs or crashes due to the (foreman) Drillbit running out of
> Heap memory.
> This is because the Web client translates the resultset into a massive object
> in the heap-space and tries to send that back to the browser. This results in
> the VM thread actively trying to perform GC if the memory is not sufficient.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
