Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16435
Change subject: [WIP] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators ...................................................................... [WIP] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators Catalogd RPC response contains the updated catalog objects in a full form. However, LocalCatalog coordinators just need the object names for invalidations. It's a waste of space to send the full catalog objects to LocalCatalog coordinators. On the other hand, there is a risk of OOM due to hitting the Java array limit when serializing a table that has a huge metadata footprint. This patch refactor the catalogd RPC responses to only send back invalidations in need. To distinguish between legacy and LocalCatalog coordinators, a new field, want_minimal_response, is introduced in TCatalogServiceRequestHeader which is the header for most of the Catalogd RPC requests (e.g. TDdlExecRequest, TUpdateCatalogRequest and TResetMetadataRequest). LocalCatalog coordinators will set this field to true. When adding updated catalog objects to the response, catalogd will add invalidations which only contain the object names (e.g. db name, table name). Note that function objects are small so are ignored in this optimization. Tests: - Run test_ddl.py in both legacy catalog mode and local catalog mode. - TODO: figure out how to add test coverage for this Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38 --- M be/src/exec/catalog-op-executor.cc M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java 10 files changed, 206 insertions(+), 138 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16435/1 -- To view, visit http://gerrit.cloudera.org:8080/16435 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38 Gerrit-Change-Number: 16435 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
