[
https://issues.apache.org/jira/browse/IMPALA-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18074861#comment-18074861
]
ASF subversion and git services commented on IMPALA-14806:
----------------------------------------------------------
Commit 71c63ed6132c73e4cf0907014d3f2fd8f1e9a6ee in impala's branch
refs/heads/master from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=71c63ed61 ]
IMPALA-14806: Cache IcebergContentFileStore
This change was split out from larger change
https://gerrit.cloudera.org/#/c/24049/
With this patch Iceberg file descriptors are cached in
IcebergFileContentStore instead of TPartialTableInfo.
TPartialInfo is still cached, but without file info.
Based on experiments this both makes mem need smaller and
planning much faster by not needing to convert TPartialTableInfo
to IcebergFileContentStore in each query. The drawback
is that +1 GetPartialCatalogObject RPC is needed to
fetch an Iceberg table from catalog.
Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.
Mem:
Before:
558 MB org.apache.impala.thrift.TPartialTableInfo
After:
449 MB org.apache.impala.catalog.IcebergFileContentStore
3 MB org.apache.impala.thrift.TPartialTableInfo
Speed:
DESCRIBE t: 1.7s->10ms
EXPLAIN SELECT * FROM t: 5.5s->4.7s
EXPLAIN SELECT * FROM t WHERE part_col=100: 1.5s->0.7s
Change-Id: Ic93eeb3a90a280a7138672affd71ac6f87df71a1
Reviewed-on: http://gerrit.cloudera.org:8080/24177
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Cache IcebergContentFileStore instead of TPartialTableInfo
> ----------------------------------------------------------
>
> Key: IMPALA-14806
> URL: https://issues.apache.org/jira/browse/IMPALA-14806
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
> Reporter: Csaba Ringhofer
> Assignee: Csaba Ringhofer
> Priority: Critical
>
> Based on measurements in https://gerrit.cloudera.org/#/c/24049
> IcebergContentFileStore is small and makes planning much faster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
