Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13772
Change subject: [WIP] IMPALA-8732: Prototype for serialized descriptor table ...................................................................... [WIP] IMPALA-8732: Prototype for serialized descriptor table In IMPALA-8732, there is contention in tcmalloc when sending the ExecQueryFInstances messages for a query referencing a large number of partitions. This is because each thread in the ExecEnv::exec_rpc_thread_pool_ is making a copy of the TQueryCtx, which contains the TDescriptorTable and a large map of THdfsPartition objects. Every thread in the exec_rpc_thread_pool_ is doing this simultaneously. The threads are copying this structure, but this structure is the same across all messages for this query. Copying a large map of THdfsPartition objects is wasteful, especially considering that the coordinator does not need to access any of this information before sending it out to executors. This changes TQueryCtx to contain a TDescriptorTableSerialized, which is a binary blob containing the serialized form of TDescriptorTable. This is serialized in the frontend and passed directly through to executors. The old unserialized TDescriptorTable form is maintained to enable frontend planner tests (which use incomplete structures lacking some required fields and cannot be serialized). Testing: - Core end to end tests pass - Running an exhaustive set of tests Change-Id: I458aa62dd4d1e4e4a7b1869a604623a69f3b2d9a --- M be/src/runtime/coordinator.cc M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/query-state.cc M common/thrift/Descriptors.thrift M common/thrift/ImpalaInternalService.thrift M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 9 files changed, 90 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/13772/1 -- To view, visit http://gerrit.cloudera.org:8080/13772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I458aa62dd4d1e4e4a7b1869a604623a69f3b2d9a Gerrit-Change-Number: 13772 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com>
