[
https://issues.apache.org/jira/browse/IMPALA-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117005#comment-17117005
]
Sahil Takiar commented on IMPALA-9757:
--------------------------------------
Did some investigation, and I don't think this is actually a bug in the code,
just the way the tests are run.
It looks like the hs2-http connections are just timing out and getting cleaned
up by the session maintenance threads:
{code}
logs/ee_tests/impalad.impala-ec2-centos74-m5-4xlarge-ondemand-13fb.vpc.cloudera.com.jenkins.log.INFO.20200522-193649.23127:I0523
02:20:07.126468 13972 impala-server.cc:1079]
28419d6d1127487e:0e10689800000000] Registered query
query_id=28419d6d1127487e:0e10689800000000
session_id=f94437c282d6c1c3:204ad55e51c6d6a9
logs/ee_tests/impalad.impala-ec2-centos74-m5-4xlarge-ondemand-13fb.vpc.cloudera.com.jenkins.log.INFO.20200522-193649.23127:I0523
03:20:37.240694 24158 impala-server.cc:2229] Closing session:
f94437c282d6c1c3:204ad55e51c6d6a9, user: jenkins, because it no longer has any
open connections. The last connection was closed at: 2020-05-23 02:20:37.158
logs/ee_tests/impalad.impala-ec2-centos74-m5-4xlarge-ondemand-13fb.vpc.cloudera.com.jenkins.log.INFO.20200522-193649.23127:I0523
03:48:52.916052 5569 impala-server.cc:1403] GetSessionState(): Invalid
session id: f94437c282d6c1c3:204ad55e51c6d6a9
logs/ee_tests/impalad.impala-ec2-centos74-m5-4xlarge-ondemand-13fb.vpc.cloudera.com.jenkins.log.INFO.20200522-193649.23127:I0523
03:48:53.082515 5611 impala-server.cc:1334] Closing session:
f94437c282d6c1c3:204ad55e51c6d6a9
logs/ee_tests/impalad.impala-ec2-centos74-m5-4xlarge-ondemand-13fb.vpc.cloudera.com.jenkins.log.INFO.20200522-193649.23127:I0523
03:48:53.082520 5611 impala-server.cc:1351] CloseSessionInternal(): Invalid
session id: f94437c282d6c1c3:204ad55e51c6d6a9
{code}
Looking through the code, the session gets invalidated if it is not used for
more than 15 minutes. It looks like in some exhaustive tests this can happen.
The hs2-http connection is opened when the class is created (see
impala_test_suite.py), a few tests run that use the hs2-http, then some other
tests run that don't use hs2-http. 15 minutes elapse and the hs2-http
connection is closed, and then future attempts to use the hs2-http connection
fail. It looks like this only happens in exhaustive tests because we typically
only add the client protocol as a test dimension in exhaustive builds.
A simple fix might be to just bump the session timeout for tests.
> Test failures with HiveServer2Error: Invalid session id
> -------------------------------------------------------
>
> Key: IMPALA-9757
> URL: https://issues.apache.org/jira/browse/IMPALA-9757
> Project: IMPALA
> Issue Type: Test
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Labels: broken-build, flaky
>
> Only seen once so far on an exhaustive build. It's not clear if the
> "HiveServer2Error: Invalid session id" error is specific to this test or not.
> {code:java}
> query_test.test_queries.TestQueries.test_inline_view[protocol: hs2-http |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none] (from pytest)
> Error Message
> query_test/test_queries.py:104: in test_inline_view
> self.run_test_case('QueryTest/inline-view', vector)
> common/impala_test_suite.py:567: in run_test_case table_format_info,
> use_db, pytest.config.option.scale_factor) common/impala_test_suite.py:782:
> in change_database impala_client.execute(query)
> common/impala_connection.py:331: in execute handle =
> self.execute_async(sql_stmt, user) common/impala_connection.py:354: in
> execute_async self.__cursor.execute_async(sql_stmt,
> configuration=self.__query_options)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:375:
> in execute_async self._execute_async(op)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:394:
> in _execute_async operation_fn()
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:372:
> in op run_async=True)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1096:
> in execute return self._operation('ExecuteStatement', req)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1026:
> in _operation resp = self._rpc(kind, request)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:994:
> in _rpc err_if_rpc_not_ok(response)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:748:
> in err_if_rpc_not_ok raise HiveServer2Error(resp.status.errorMessage) E
> HiveServer2Error: Invalid session id: 3345279d9b2e75ab:3aef93f7a80d7d8a
> Stacktrace
> query_test/test_queries.py:104: in test_inline_view
> self.run_test_case('QueryTest/inline-view', vector)
> common/impala_test_suite.py:567: in run_test_case
> table_format_info, use_db, pytest.config.option.scale_factor)
> common/impala_test_suite.py:782: in change_database
> impala_client.execute(query)
> common/impala_connection.py:331: in execute
> handle = self.execute_async(sql_stmt, user)
> common/impala_connection.py:354: in execute_async
> self.__cursor.execute_async(sql_stmt, configuration=self.__query_options)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:375:
> in execute_async
> self._execute_async(op)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:394:
> in _execute_async
> operation_fn()
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:372:
> in op
> run_async=True)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1096:
> in execute
> return self._operation('ExecuteStatement', req)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1026:
> in _operation
> resp = self._rpc(kind, request)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:994:
> in _rpc
> err_if_rpc_not_ok(response)
> /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:748:
> in err_if_rpc_not_ok
> raise HiveServer2Error(resp.status.errorMessage)
> E HiveServer2Error: Invalid session id: 3345279d9b2e75ab:3aef93f7a80d7d8a
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
