[jira] [Created] (IMPALA-8389) e2e custom cluster testsuite does not respect cluster_size when impala_log_dir present
radford nguyen created IMPALA-8389:
--
Summary: e2e custom cluster testsuite does not respect
cluster_size when impala_log_dir present
Key: IMPALA-8389
URL: https://issues.apache.org/jira/browse/IMPALA-8389
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.2.0
Reporter: radford nguyen
h3. Brief
CustomClusterTestSuite always waits for 3 daemons on startup instead of
{{cluster_size}} daemons when {{impala_log_dir}} is specified.
h3. Description
The {{@CustomClusterTestSuite.withArgs}} decorator allows a user to specify a
custom cluster size for the test case being decorated. However, when this
option is specified in conjunction with {{impala_log_dir}}, it will fail to
wait for the correct number of daemons if any value other than
{{DEFAULT_CLUSTER_SIZE}} is used.
The root cause is the difference in how the cluster is started with and without
{{impala_log_dir}}:
[https://github.com/apache/impala/blob/3.2.0/tests/common/custom_cluster_test_suite.py#L147]
h3. To Reproduce:
* add {{cluster_size=5}} to decorator of test_grant_revoke in
tests/authorization/test_ranger.py
* $ impala-py.test tests/authorization/test_ranger.py
* observe pass
* add {{impala_log_dir=whatev}} to decorator of test_grant_revoke
* $ impala-py.test tests/authorization/test_ranger.py
* observe fail during cluster startup:
** 2019-04-04 14:25:54,140 INFO MainThread: Waiting for
num_known_live_backends=3. Current value: 5
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-8389) e2e custom cluster testsuite does not respect cluster_size when impala_log_dir present
[
https://issues.apache.org/jira/browse/IMPALA-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
radford nguyen resolved IMPALA-8389.
Resolution: Fixed
> e2e custom cluster testsuite does not respect cluster_size when
> impala_log_dir present
> --
>
> Key: IMPALA-8389
> URL: https://issues.apache.org/jira/browse/IMPALA-8389
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: radford nguyen
>Assignee: radford nguyen
>Priority: Minor
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> h3. Brief
> CustomClusterTestSuite always waits for 3 daemons on startup instead of
> {{cluster_size}} daemons when {{impala_log_dir}} is specified.
> h3. Description
> The {{@CustomClusterTestSuite.withArgs}} decorator allows a user to specify a
> custom cluster size for the test case being decorated. However, when this
> option is specified in conjunction with {{impala_log_dir}}, it will fail to
> wait for the correct number of daemons if any value other than
> {{DEFAULT_CLUSTER_SIZE}} is used.
> The root cause is the difference in how the cluster is started with and
> without {{impala_log_dir}}:
> [https://github.com/apache/impala/blob/3.2.0/tests/common/custom_cluster_test_suite.py#L147]
> h3. To Reproduce:
> * add {{cluster_size=5}} to decorator of test_grant_revoke in
> tests/authorization/test_ranger.py
> * $ impala-py.test tests/authorization/test_ranger.py
> * observe pass
> * add {{impala_log_dir=whatev}} to decorator of test_grant_revoke
> * $ impala-py.test tests/authorization/test_ranger.py
> * observe fail during cluster startup:
> ** 2019-04-04 14:25:54,140 INFO MainThread: Waiting for
> num_known_live_backends=3. Current value: 5
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-8473) Refactor lineage publication mechanism to allow for different consumers
radford nguyen created IMPALA-8473: -- Summary: Refactor lineage publication mechanism to allow for different consumers Key: IMPALA-8473 URL: https://issues.apache.org/jira/browse/IMPALA-8473 Project: IMPALA Issue Type: Improvement Components: Backend, Frontend Reporter: radford nguyen Impetus for this change is to allow lineage to be consumed by Altus via Kafka. Approach should be similar to that of choosing authorization provider, where the publication strategy can be chosen at runtime via configuration flag(s). Scope of this ticket is to move lineage publication to the fe and add appropriate hooks that a user can implement -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8571) Make query-hook-execution more robust and observable
radford nguyen created IMPALA-8571: -- Summary: Make query-hook-execution more robust and observable Key: IMPALA-8571 URL: https://issues.apache.org/jira/browse/IMPALA-8571 Project: IMPALA Issue Type: Improvement Components: Frontend Reporter: radford nguyen Assignee: radford nguyen Placeholder: description coming soon -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8572) Move query hook execution to before query unregistration
radford nguyen created IMPALA-8572: -- Summary: Move query hook execution to before query unregistration Key: IMPALA-8572 URL: https://issues.apache.org/jira/browse/IMPALA-8572 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: radford nguyen -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8573) Implement timeout for query hook execution
radford nguyen created IMPALA-8573: -- Summary: Implement timeout for query hook execution Key: IMPALA-8573 URL: https://issues.apache.org/jira/browse/IMPALA-8573 Project: IMPALA Issue Type: Improvement Reporter: radford nguyen Placeholder: description coming soon -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8574) Review test coverage for query hook feature
radford nguyen created IMPALA-8574: -- Summary: Review test coverage for query hook feature Key: IMPALA-8574 URL: https://issues.apache.org/jira/browse/IMPALA-8574 Project: IMPALA Issue Type: Improvement Reporter: radford nguyen Assignee: radford nguyen Placeholder: description coming soon -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8576) Pass lineage object instead of string to query hook
radford nguyen created IMPALA-8576: -- Summary: Pass lineage object instead of string to query hook Key: IMPALA-8576 URL: https://issues.apache.org/jira/browse/IMPALA-8576 Project: IMPALA Issue Type: Improvement Components: Backend, Frontend Reporter: radford nguyen Placeholder: description coming soon -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8573) Implement timeout for query hook execution
[ https://issues.apache.org/jira/browse/IMPALA-8573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] radford nguyen resolved IMPALA-8573. Resolution: Duplicate > Implement timeout for query hook execution > -- > > Key: IMPALA-8573 > URL: https://issues.apache.org/jira/browse/IMPALA-8573 > Project: IMPALA > Issue Type: Sub-task >Reporter: radford nguyen >Priority: Major > > Placeholder: description coming soon -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8914) Collect and publish QueryEventHook metrics
radford nguyen created IMPALA-8914:
--
Summary: Collect and publish QueryEventHook metrics
Key: IMPALA-8914
URL: https://issues.apache.org/jira/browse/IMPALA-8914
Project: IMPALA
Issue Type: Improvement
Components: Backend, Frontend
Reporter: radford nguyen
Metrics around {{QueryEventHook}} execution are currently not captured. It
would help debug/diagnosis to capture such metrics as:
* # of times hook execution has been rejected (due to full work queue)
* # of times hook execution has timed out
* # of times hook execution has exceptioned
* # of times hook execution has been requested
* avg time hook execution has spent queued up
* avg time hook execution has taken to execute
All metrics above should be further qualified by the hook implementation class.
The following naming convention is suggested (in respective order):
* query-event-hook.${hook}.num-execution-rejections
* query-event-hook.${hook}.num-execution-timeouts
* query-event-hook.${hook}.num-execution-exceptions
* query-event-hook.${hook}.num-execution-submissions
* query-event-hook.${hook}.mean-queued-time
* query-event-hook.${hook}.mean-execution-time
where ${hook} is the fully-qualified method name of the hook method, e.g.
com.acme.impala.hooks.MyImpalaHook.onQueryComplete
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
