Hey Prabhakar, can you run exactly the same test as below after setting
drill.exec.http.memory.heap.failure.threshold = 2
in your drill-override.conf? I have a theory that we're carrying some
memory usage limiting code that we no longer need since we gained
streaming HTTP results, and that might be interacting badly with the
Java garbage collector. Alternatively, and I sincerely hope it's not the
case, we are genuinely leaking heap memory somewhere and need to find
it. If we get a favourable result from your testing I think we should
consider incorporating an extra fix in 1.20.2 that removes the home
grown memory usage limiting.
Footnote: the value of 2 above (200% of the maximum heap size) is not
meant to be a sensible fraction but was chosen only to completely
circumvent the mentioned memory usage limiting logic in Drill.
On 2022/07/22 08:50, Prabhakar Bhosale wrote:
Hi Team,
I am Running the drill in embedded mode. The drill version is 1.20.1.
After running multiple queries from WEB UI which opens 8000 parquet
compressed files it gives me the below error. I am running queries one
after another. My understanding is that once the query is completed
successfully or failed then it should release the memory. So the system
should not give this error if I am executing the queries one at a time.
I have not changed any memory settings for drill. Any suggestions?
warning UserException : RESOURCE ERROR: There is not enough heap memory to
run this query using the web interface.
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is
not enough heap memory to run this query using the web interface. Please
try a query with fewer columns or with a filter or limit condition to limit
the data returned. You can also try an ODBC/JDBC client. [Error Id:
c63dda94-071a-4c79-953c-fab956418cd8 ]
