James Turton created DRILL-8283:
-----------------------------------
Summary: Implement a configurable file listing size limit
Key: DRILL-8283
URL: https://issues.apache.org/jira/browse/DRILL-8283
Project: Apache Drill
Issue Type: Improvement
Components: Storage - Other
Affects Versions: 1.20.2
Reporter: James Turton
Assignee: James Turton
Fix For: 1.20.3
Currently a malicious, or merely an unwitting user can crash their Drill
foreman by sending
{code:java}
select * from dfs.huge_workspace limit 10
{code}
causing the query planner to recurse over every file in huge_workspace and
culminating in
{code:java}
2022-08-09 15:13:22,251 [1d0da29f-e50c-fd51-43d9-8a5086d52c4e:foreman] ERROR
o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, exiting.
Information message: Unable to handle out of memory condition in
Foreman.java.lang.OutOfMemoryError: null {code}
if there are enough files in huge_workspace. A SHOW FILES command can produce
the same effect. This issue proposes a new BOOT option named
drill.exec.storage.file.max_listing_size with a default value of, say 10 000.
If a file listing task exceeds this limit then the current operation is
terminated with a UserException and runaway resource usage is prevented.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
