[ https://issues.apache.org/jira/browse/DRILL-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arina Ielchiieva updated DRILL-7338: ------------------------------------ Labels: doc-impacting ready-to-commit (was: ready-to-commit) > REST API calls to Drill fail due to insufficient heap memory > ------------------------------------------------------------ > > Key: DRILL-7338 > URL: https://issues.apache.org/jira/browse/DRILL-7338 > Project: Apache Drill > Issue Type: Bug > Components: Web Server > Affects Versions: 1.15.0 > Reporter: Aditya Allamraju > Assignee: Kunal Khatua > Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.17.0 > > > Drill queries that use REST API calls have started failing(given below) after > recent changes. > {code:java} > RESOURCE ERROR: There is not enough heap memory to run this query using the > web interface. > Please try a query with fewer columns or with a filter or limit condition to > limit the data returned. > You can also try an ODBC/JDBC client.{code} > They were running fine earlier as the ResultSet returned was just few rows. > These queries now fail for even very small resultSets( < 10rows). > Investigating the issue revealed that we introduced a check to limit the Heap > usage. > The Wrapper code from > *_exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java_* > that throws this error, i see certain issues. It does seem we use a > threshold of *85%* of heap usage before throwing that warning and exiting the > query. > > {code:java} > public class QueryWrapper { > private static final org.slf4j.Logger logger = > org.slf4j.LoggerFactory.getLogger(QueryWrapper.class); > // Heap usage threshold/trigger to provide resiliency on web server for > queries submitted via HTTP > private static final double HEAP_MEMORY_FAILURE_THRESHOLD = 0.85; > ... > private static MemoryMXBean memMXBean = ManagementFactory.getMemoryMXBean(); > ... > // Wait until the query execution is complete or there is error submitting > the query > logger.debug("Wait until the query execution is complete or there is > error submitting the query"); > do { > try { > isComplete = webUserConnection.await(TimeUnit.SECONDS.toMillis(1)); > //periodically timeout 1 sec to check heap > } catch (InterruptedException e) {} > usagePercent = getHeapUsage(); > if (usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD) { > nearlyOutOfHeapSpace = true; > } > } while (!isComplete && !nearlyOutOfHeapSpace); > {code} > By using above check, we unintentionally invited all those issues that happen > with Java’s Heap usage. JVM does try to make maximum usage of HEAP until > Minor or Major GC kicks in i.e GC kicks after there is no more space left in > heap(eden or young gen). > The workarounds i can think of in order to resolve this issue are: > # Remove this check altogether so we know why it is filling up Heap. > # Advise the users to stop using REST for querying data.(We did this > already). *But not all users may not be happy with this suggestion.* There > could be few dynamic applications(dashboard, monitoring etc). > # Make the threshold high enough so that GC kicks in much better. > If not above options, we have to tune the Heap sizes of drillbit. A quick fix > would be to increase the threshold from 85% to 100%(option-3 above). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)