Hello Quanlong Huang, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24711
to look at the new patch set (#7).
Change subject: IMPALA-15234: Add a way to clear the HBO cache
......................................................................
IMPALA-15234: Add a way to clear the HBO cache
HBO keeps historical runs in a per-coordinator cache inside the
frontend. Once a run that no longer describes the data is stored, for
example after a table is overwritten with a very different
distribution, it keeps influencing plans until it is evicted.
End-to-end tests have the same problem in reverse: a run stored by an
earlier query can satisfy a later assertion.
Add CacheBackend.clear(), reachable from the backend through a new
clearHboStats() JNI entry point, and expose it as the
/clear_hbo_stats WebUI endpoint. As the JIRA suggests, this drops the
cache as a whole: entries are keyed by hash strings, so the ones
belonging to a given table cannot be identified.
The clear covers what is in the cache when it runs. Writers are not
serialized against anything - writePlanNodeStats() reads an entry,
edits its run list and puts it back, which is the race its own TODO
already describes - so a query unregistering at that moment can store a
run after the clear returned. That is said next to clearCache() rather
than worked around, since holding it off means taking on the writer
races the class starts with.
The cache lives in the frontend of each coordinator, so the endpoint
clears the coordinator that serves the request. Require POST so the
mutation goes through the existing Webserver CSRF checks. GET returns
an error without changing the cache, and the end-to-end test covers
both paths.
Testing:
- InMemoryCacheBackendTest
- TestHBO::test_clear_hbo_cache
- python3 -m py_compile tests/common/impala_service.py
tests/query_test/test_hbo.py
- git diff --check
Change-Id: Ib58305e96f05ba49eb03d935765620915964308a
Assisted-by: claude-opus-5 (Claude Code)
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-http-handler.h
M fe/src/main/java/org/apache/impala/service/CacheBackend.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/main/java/org/apache/impala/service/InMemoryCacheBackend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A fe/src/test/java/org/apache/impala/service/InMemoryCacheBackendTest.java
M tests/common/impala_service.py
M tests/query_test/test_hbo.py
11 files changed, 186 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/24711/7
--
To view, visit http://gerrit.cloudera.org:8080/24711
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib58305e96f05ba49eb03d935765620915964308a
Gerrit-Change-Number: 24711
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
