[
https://issues.apache.org/jira/browse/IMPALA-12715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18068154#comment-18068154
]
Quanlong Huang commented on IMPALA-12715:
-----------------------------------------
Recently saw this failed in two builds:
[https://jenkins.impala.io/job/ubuntu-20.04-from-scratch/8445]
[https://jenkins.impala.io/job/ubuntu-20.04-from-scratch/8447]
The test runs a DESCRIBE to trigger metadata loading on the table. Then expects
the owner info is loaded and the following INVALIDATE command should pass.
{code:python}
# Run a query to trigger metadata loading on the table
self.execute_query_expect_success(
non_admin_client, "describe functional.alltypestiny")
...
# INVALIDATE METADATA <table> is allowed since 'user' is detected as the
owner.
self.execute_query_expect_success(
non_admin_client, "invalidate metadata functional.alltypestiny") #
<-- This fails
{code}
This is true for the legacy catalog mode, i.e., when the table metadata is
loaded in catalogd, coordinator that triggers the loading also get its metadata
loaded. However, for the local catalog mode, there is a race issue that the
loaded metadata could be invalidated by statestore updates (IMPALA-14856).
There are coordinator logs indicating this:
{noformat}
I20260321 19:17:47.899945 2610038 Frontend.java:2483]
dc4ea98f2f0f63d0:fff6189800000000] Analyzing query: invalidate metadata
functional.alltypestiny db: default
...
I20260321 19:17:47.900617 2609929 CatalogdMetaProvider.java:1818] Invalidated
objects in cache: [table functional.alltypestiny, TABLE_LIST for DB functional]
W20260321 19:17:47.900683 2610038 LocalTable.java:237]
dc4ea98f2f0f63d0:fff6189800000000] Owner of functional.alltypestiny is unknown
due to msTable is unloaded
I20260321 19:17:47.900691 2609929 impala-server.cc:2351] Catalog topic update
applied with version: 2093 new min catalog object version: 3
...
I20260321 19:17:47.901367 2610038 jni-util.cc:335]
dc4ea98f2f0f63d0:fff6189800000000]
org.apache.impala.authorization.AuthorizationException: User 'ubuntu' does not
have privileges to execute 'INVALIDATE METADATA/REFRESH' on:
functional.alltypestiny{noformat}
Thread 2609929 was applying catalog updates from statestore. It invalidated the
table meta just before INVALIDATE used the owner info. So the INVALIDATE thread
complained that "Owner of ... is unknown due to msTable is unloaded" and
finally failed.
To deflake the test, we can use a DDL with sync_ddl=true to trigger metadata
loading in catalogd side and make sure coordinator applies the statestore
update. Then run the DESCRIBE to make sure the table meta is also loaded in
coordinator side.
Uploaded a patch for review: http://gerrit.cloudera.org:8080/24135
CC [~mszjat], [~csringhofer]
> Failing TestRanger.test_allow_metadata_update_local_catalog test
> ----------------------------------------------------------------
>
> Key: IMPALA-12715
> URL: https://issues.apache.org/jira/browse/IMPALA-12715
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.4.0
> Reporter: Tamas Mate
> Assignee: Quanlong Huang
> Priority: Critical
>
> Two internal JDK17 core builds have failed with the bellow
> TestRanger.test_allow_metadata_update_local_catalog failure. Standard error
> shows connection failures.
> {code:none}
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: <class
> 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AuthorizationException: User
> 'jenkins' does not have privileges to execute 'INVALIDATE METADATA/REFRESH'
> on: functional.alltypestiny
> {code}
> *Stacktrace*
> {code:none}
> authorization/test_ranger.py:1582: in test_allow_metadata_update_local_catalog
> self.__test_allow_catalog_cache_op_from_masked_users(unique_name)
> authorization/test_ranger.py:1615: in
> __test_allow_catalog_cache_op_from_masked_users
> non_admin_client, "invalidate metadata functional.alltypestiny",
> user=user)
> common/impala_test_suite.py:944: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:952: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:1069: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:218: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:367: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:361: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:524: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E MESSAGE: AuthorizationException: User 'jenkins' does not have privileges
> to execute 'INVALIDATE METADATA/REFRESH' on: functional.alltypestiny
> {code}
> *Standard error:*
> {code:none}
> -- 2024-01-12 01:49:18,544 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos79-m6i-4xlarge-ondemand-1347.vpc.cloudera.com:25002
> -- 2024-01-12 01:49:18,546 INFO MainThread: num_known_live_backends has
> reached value: 3
> SET
> client_identifier=authorization/test_ranger.py::TestRanger::()::test_allow_metadata_update_local_catalog;
> -- connecting to: localhost:21000
> -- 2024-01-12 01:49:18,546 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/thrift-0.16.0-p6/python/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- connecting to localhost:21050 with impyla
> -- 2024-01-12 01:49:18,546 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/thrift-0.16.0-p6/python/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2024-01-12 01:49:18,654 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2024-01-12 01:49:18,671 INFO MainThread: Closing active operation
> -- connecting to localhost:11050 with impyla
> SET
> client_identifier=authorization/test_ranger.py::TestRanger::()::test_allow_metadata_update_local_catalog;
> -- connecting to: localhost:21000
> -- 2024-01-12 01:49:18,677 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/thrift-0.16.0-p6/python/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> SET
> client_identifier=authorization/test_ranger.py::TestRanger::()::test_allow_metadata_update_local_catalog;
> -- connecting to: localhost:21000
> -- 2024-01-12 01:49:18,678 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/thrift-0.16.0-p6/python/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2024-01-12 01:49:18,698 INFO MainThread: Added column masking policy
> 254
> -- executing against localhost:21000
> invalidate metadata functional.alltypestiny;
> -- executing against localhost:21000
> describe functional.alltypestiny;
> -- 2024-01-12 01:49:19,137 INFO MainThread: Started query
> e34f407b57a251be:b7f9db8400000000
> -- 2024-01-12 01:49:19,141 INFO MainThread: Expected log lines could not
> be found, sleeping before retrying: Expected 1 lines in file
> /data0/jenkins/workspace/impala-cdw-master-staging-core-jdk17/repos/Impala/logs/custom_cluster_tests/catalogd.impala-ec2-centos79-m6i-4xlarge-ondemand-1347.vpc.cloudera.com.jenkins.log.INFO.20240112-014909.29666
> matching regex 'Loaded metadata for: functional.alltypestiny', but found 0
> lines. Last line was:
> I0112 01:49:14.127092 30152 MetastoreEventsProcessor.java:1023] Latest event
> in HMS: id=63862, time=1705052926. Last synced event: id=63862,
> time=1705052926.
> -- executing against localhost:21000
> invalidate metadata functional.alltypestiny;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
