Hello Bharath Vissapragada, Andrew Sherman, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13748 to look at the new patch set (#27). Change subject: IMPALA-8571: harden QueryEventHook execution ...................................................................... IMPALA-8571: harden QueryEventHook execution This commit hardens guarantees around QueryEventHook execution by adding the following features: *hook execution* Query event hooks are still executed in a thread-pool, but you can now configure the type of (Java) threads used to execute them via the newly-added backend flag `query_event_hook_use_daemon_threads`, which takes a true/false value. See Java Thread.setDaemon(boolean) for what a "daemon thread" entails. *hook timeout/cancellation* A timeout for hook execution can be configured through the backend flag `query_event_hook_timeout_s`, which specified a timeout value in seconds. If a hook has not completed execution within this timeout (measured from hook submission, not execution) then the hook task will be cancelled in order to free up resources. *hook rejection* The hook execution engine now has a fixed-capacity work queue whose capacity can be configured through the backend flag `query_event_hook_queue_capacity`. This queue is used to store hook tasks that are submitted when there are no free threads available for hook execution. All hook tasks submitted when the queue is at capacity will be rejected and logged without affecting the result of the query. *hook performance metrics* Hook performance metrics are captured by the frontend, but not yet published to the backend and are therefore not user-visible. See IMPALA-8914 for implementation of this feature. Testing: - added unit tests for new features - re-ran existing E2E tests Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 --- M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java A fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java 11 files changed, 843 insertions(+), 114 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/27 -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 27 Gerrit-Owner: radford nguyen <radford.ngu...@gmail.com> Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@apache.org> Gerrit-Reviewer: Fredy Wijaya <fre...@apache.org> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: radford nguyen <radford.ngu...@gmail.com>