[ https://issues.apache.org/jira/browse/IMPALA-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Quanlong Huang updated IMPALA-9703: ----------------------------------- Labels: catalog-2024 (was: ) > Skip loading partition meta and file meta for PB scale tables > ------------------------------------------------------------- > > Key: IMPALA-9703 > URL: https://issues.apache.org/jira/browse/IMPALA-9703 > Project: IMPALA > Issue Type: New Feature > Components: Catalog > Reporter: Quanlong Huang > Priority: Critical > Labels: catalog-2024 > > PB scale tables that have >100K partitions may hit catalog limitations. > Caching all the partitions is also a waste since usually only few of them are > required. Queries scanning all partitions probably fail with resource > limitation errors so it's not in our consideration. > This JIRA tracks the work to skip caching partition meta of a table. Catalogd > will only cache the HmsTable object and partition list (partition names, e.g. > "p1=a/p2=b" and internal partition ids generated by Impala). Coordinators > fetch the partition meta on-demand when compiling queries. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org