Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24049
to look at the new patch set (#2).
Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................
POC: LocalIcebergTable loads files in coordinator
If load_iceberg_files_in_coordinator=true, load files on
the coordinator for Iceberg tables instead of getting from
the catalogd. This is inefficient at the moment as catalogd
still loads the files but not really uses them. The long
term goal is to load only minimal info for the table on
catalogd side.
Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex pair. This would be
suitable for REST catalog too if the key contained
snapshot ID instead of catalog version.
Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- Size of cache objects seem to decrease:
~543->~431 MB for 1M file table
- Plans look faster due to skipping construction of
IcebergFileContentStore (DESCRIBE 1M file table 0.5s->1ms)
- Probably needs to worry less about inconsistant metadata
exceptions, as the old file list remains loadable event after
catalogd updated to a newer version.
Cons:
- Initial table loading is ~doubled as both the catalog and
the coordinator need to load the files.
- No incremental loading, any catalog version change leads
the reloaiding the files from scratch on coordinator side.
Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M
fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
13 files changed, 161 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/2
--
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
