[ https://issues.apache.org/jira/browse/IMPALA-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Smith updated IMPALA-12161: ----------------------------------- Component/s: Frontend (was: fe) > Add query hints for memory limits > --------------------------------- > > Key: IMPALA-12161 > URL: https://issues.apache.org/jira/browse/IMPALA-12161 > Project: IMPALA > Issue Type: New Feature > Components: Frontend > Affects Versions: Impala 4.1.2 > Reporter: Peter Ebert > Priority: Major > > Support for a query hint to specify memory limits would make implementing > memory limits in applications easier where multiple queries need different > memory limits but are run through the same connection, e.g.: {{/* > +mem_limit=4g */ select * from table}} > --- > Session settings for mem_limit are critical for getting large queries to > avoid OOM when using admission control pools (for multi tenancy) and the > memory estimation happens to severely under-estimate (raising the minimum > memory for the pool would also reduce concurrency). However, for > applications that run multiple queries through a single connection (e.g. data > visualization tools) this pattern is cumbersome: > # Set the memory limit > # Run the large query > # Reset the memory limit > # Resume other queries with good estimates > Due to this, some data visualization tools only support session settings at > the connection level. If you need GBs for your worst query and MBs for your > average query, this greatly limits the concurrency in that pool using a 'one > size fits all' memory limit or it requires maintaining many connections of > varying size memory limits to attempt to use memory efficiently. > In one production use case this would limit concurrency by 10x or more. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org