[
https://issues.apache.org/jira/browse/IMPALA-12715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18069034#comment-18069034
]
ASF subversion and git services commented on IMPALA-12715:
----------------------------------------------------------
Commit 957c398ad08930878b830a1eb1c1e6138a2a6929 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=957c398ad ]
IMPALA-12715: deflake test_allow_metadata_update_local_catalog
The test expects DESCRIBE on an unloaded table can load the table meta
in coordinator side. However, this can't be guaranteed in the local
catalog mode since catalog updates(invalidations) from statestore could
invalidate the cache (IMPALA-14856).
The test is flaky since DESCRIBE uses the table meta twice, one in
analyze phase (StmtMetadataLoader) and the other in execution
(Frontend.doDescribeTable()). If the statestore update arrives between
them, the second usage can load the table meta again so the cache item
exists. However, if the statestore update arrives after the second
usage, the cache item is removed so the test fails.
This patch deflakes the test by using REFRESH on the unloaded table to
trigger metadata loading in catalogd side. Run it with sync_ddl=true to
make sure coordinator processes the statestore update. Then run a
DESCRIBE to make sure the table meta is loaded in coordinator side.
Tests:
- ran the test locally 200 times.
- ran test_ranger.py
Change-Id: I538c30bf4d1439108bd1cd1cb64208281974c1f6
Reviewed-on: http://gerrit.cloudera.org:8080/24135
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
> Failing TestRanger.test_allow_metadata_update_local_catalog test
> ----------------------------------------------------------------
>
> Key: IMPALA-12715
> URL: https://issues.apache.org/jira/browse/IMPALA-12715
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.4.0
> Reporter: Tamas Mate
> Assignee: Quanlong Huang
> Priority: Critical
>
> Two internal JDK17 core builds have failed with the bellow
> TestRanger.test_allow_metadata_update_local_catalog failure. Standard error
> shows connection failures.
> {code:none}
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: <class
> 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AuthorizationException: User
> 'jenkins' does not have privileges to execute 'INVALIDATE METADATA/REFRESH'
> on: functional.alltypestiny
> {code}
> *Stacktrace*
> {code:none}
> authorization/test_ranger.py:1582: in test_allow_metadata_update_local_catalog
> self.__test_allow_catalog_cache_op_from_masked_users(unique_name)
> authorization/test_ranger.py:1615: in
> __test_allow_catalog_cache_op_from_masked_users
> non_admin_client, "invalidate metadata functional.alltypestiny",
> user=user)
> common/impala_test_suite.py:944: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:952: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:1069: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:218: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:367: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:361: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:524: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E MESSAGE: AuthorizationException: User 'jenkins' does not have privileges
> to execute 'INVALIDATE METADATA/REFRESH' on: functional.alltypestiny
> {code}
> *Standard error:*
> {code:none}
> -- 2024-01-12 01:49:18,544 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos79-m6i-4xlarge-ondemand-1347.vpc.cloudera.com:25002
> -- 2024-01-12 01:49:18,546 INFO MainThread: num_known_live_backends has
> reached value: 3
> SET
> client_identifier=authorization/test_ranger.py::TestRanger::()::test_allow_metadata_update_local_catalog;
> -- connecting to: localhost:21000
> -- 2024-01-12 01:49:18,546 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/thrift-0.16.0-p6/python/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- connecting to localhost:21050 with impyla
> -- 2024-01-12 01:49:18,546 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/thrift-0.16.0-p6/python/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2024-01-12 01:49:18,654 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2024-01-12 01:49:18,671 INFO MainThread: Closing active operation
> -- connecting to localhost:11050 with impyla
> SET
> client_identifier=authorization/test_ranger.py::TestRanger::()::test_allow_metadata_update_local_catalog;
> -- connecting to: localhost:21000
> -- 2024-01-12 01:49:18,677 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/thrift-0.16.0-p6/python/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> SET
> client_identifier=authorization/test_ranger.py::TestRanger::()::test_allow_metadata_update_local_catalog;
> -- connecting to: localhost:21000
> -- 2024-01-12 01:49:18,678 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/thrift-0.16.0-p6/python/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-jdk17/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2024-01-12 01:49:18,698 INFO MainThread: Added column masking policy
> 254
> -- executing against localhost:21000
> invalidate metadata functional.alltypestiny;
> -- executing against localhost:21000
> describe functional.alltypestiny;
> -- 2024-01-12 01:49:19,137 INFO MainThread: Started query
> e34f407b57a251be:b7f9db8400000000
> -- 2024-01-12 01:49:19,141 INFO MainThread: Expected log lines could not
> be found, sleeping before retrying: Expected 1 lines in file
> /data0/jenkins/workspace/impala-cdw-master-staging-core-jdk17/repos/Impala/logs/custom_cluster_tests/catalogd.impala-ec2-centos79-m6i-4xlarge-ondemand-1347.vpc.cloudera.com.jenkins.log.INFO.20240112-014909.29666
> matching regex 'Loaded metadata for: functional.alltypestiny', but found 0
> lines. Last line was:
> I0112 01:49:14.127092 30152 MetastoreEventsProcessor.java:1023] Latest event
> in HMS: id=63862, time=1705052926. Last synced event: id=63862,
> time=1705052926.
> -- executing against localhost:21000
> invalidate metadata functional.alltypestiny;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
