Hello Bharath Vissapragada, Todd Lipcon, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13849
to look at the new patch set (#5).
Change subject: IMPALA-8486: fix stale libCache entries in LocalCatalog mode
coordinators
......................................................................
IMPALA-8486: fix stale libCache entries in LocalCatalog mode coordinators
In LocalCatalog mode, after a function is dropped, statestored will
broadcast the update to invalidate the cached CatalogObject in each
coordinator (if they have). However, the current code path does not
trigger libCache to remove the cached JAR/SO file. If we replace the
function file in HDFS with a new one and create the function again
using the same HDFS path, the SELECT statements in other coordinators
won't trigger libCache to refresh the local cached file, so they still
use the old cached file which causes errors.
When a coordinator invalidates its cached CatalogObject of a function,
it should also mark the corresponding libCache entry as "needs refresh".
So the later usage of this function will check the last modified time of
the HDFS file and refresh it in needs. To achieve this, we have to
propagate the HDFS path of the function along with the full function
name in the minimal topic, so libCache can target the cached entry.
Note that this does not prevent the dedicated executors to have stale
libCache entries. It needs some architecture changes. We'll follow it
in IMPALA-8763.
Tests
- Re-enable test_udf_update_via_drop and test_udf_update_via_create for
LocalCatalog mode.
Change-Id: Ie4812fb8737de3ba6074ffeb9007927bfbbbaf9b
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M tests/common/skip.py
M tests/query_test/test_udfs.py
4 files changed, 13 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/13849/5
--
To view, visit http://gerrit.cloudera.org:8080/13849
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie4812fb8737de3ba6074ffeb9007927bfbbbaf9b
Gerrit-Change-Number: 13849
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
