[jira] [Updated] (IMPALA-9726) Update boilerplate in the PyPI sidebar for impala-shell supported versions
[
https://issues.apache.org/jira/browse/IMPALA-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-9726:
Description:
The following lines need to be updated to reflect that the shell now supports
python 2.7+ and 3+.
https://github.com/apache/impala/blob/master/shell/packaging/setup.py#L164-167
{noformat}
'Programming Language :: Python :: 2 :: Only',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
{noformat}
Note that this has no effect on the actual installation. The following line is
what manages that, and its value is correct for both Impala 3.4.0 and Impala
4.0:
https://github.com/apache/impala/blob/master/shell/packaging/setup.py#L138
was:
The following lines need to be updated to reflect that the shell now supports
python 2.7+ and 3+.
https://github.com/apache/impala/blob/master/shell/packaging/setup.py#L164-167
{noformat}
'Programming Language :: Python :: 2 :: Only',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
{noformat}
Note that this has no effect on the actual installation. This line is what
manages that, and its value is correct for both Impala 3.4.0 and Impala 4.0:
https://github.com/apache/impala/blob/master/shell/packaging/setup.py#L138
> Update boilerplate in the PyPI sidebar for impala-shell supported versions
> --
>
> Key: IMPALA-9726
> URL: https://issues.apache.org/jira/browse/IMPALA-9726
> Project: IMPALA
> Issue Type: Sub-task
> Components: Clients
>Affects Versions: Impala 4.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Minor
>
> The following lines need to be updated to reflect that the shell now supports
> python 2.7+ and 3+.
> https://github.com/apache/impala/blob/master/shell/packaging/setup.py#L164-167
> {noformat}
> 'Programming Language :: Python :: 2 :: Only',
> 'Programming Language :: Python :: 2.6',
> 'Programming Language :: Python :: 2.7',
> {noformat}
> Note that this has no effect on the actual installation. The following line
> is what manages that, and its value is correct for both Impala 3.4.0 and
> Impala 4.0:
> https://github.com/apache/impala/blob/master/shell/packaging/setup.py#L138
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9726) Update boilerplate in the PyPI sidebar for impala-shell supported versions
David Knupp created IMPALA-9726:
---
Summary: Update boilerplate in the PyPI sidebar for impala-shell
supported versions
Key: IMPALA-9726
URL: https://issues.apache.org/jira/browse/IMPALA-9726
Project: IMPALA
Issue Type: Sub-task
Components: Clients
Affects Versions: Impala 4.0
Reporter: David Knupp
The following lines need to be updated to reflect that the shell now supports
python 2.7+ and 3+.
https://github.com/apache/impala/blob/master/shell/packaging/setup.py#L164-167
{noformat}
'Programming Language :: Python :: 2 :: Only',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
{noformat}
Note that this has no effect on the actual installation. This line is what
manages that, and its value is correct for both Impala 3.4.0 and Impala 4.0:
https://github.com/apache/impala/blob/master/shell/packaging/setup.py#L138
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9719) Upgrade sasl-0.1.1 to 0.2.1 in Impala/shell/ext-py
[ https://issues.apache.org/jira/browse/IMPALA-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9719. - Fix Version/s: Impala 4.0 Resolution: Fixed > Upgrade sasl-0.1.1 to 0.2.1 in Impala/shell/ext-py > -- > > Key: IMPALA-9719 > URL: https://issues.apache.org/jira/browse/IMPALA-9719 > Project: IMPALA > Issue Type: Sub-task > Components: Clients >Affects Versions: Impala 4.0 >Reporter: David Knupp >Priority: Major > Fix For: Impala 4.0 > > > Needed for python 3 compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9720) Upgrade bitarray from 0.9.0 to 1.2.1 in Impala/shell/ext-py
[ https://issues.apache.org/jira/browse/IMPALA-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9720. - Fix Version/s: Impala 4.0 Resolution: Fixed > Upgrade bitarray from 0.9.0 to 1.2.1 in Impala/shell/ext-py > --- > > Key: IMPALA-9720 > URL: https://issues.apache.org/jira/browse/IMPALA-9720 > Project: IMPALA > Issue Type: Sub-task > Components: Clients >Affects Versions: Impala 4.0 >Reporter: David Knupp >Priority: Major > Fix For: Impala 4.0 > > > This is needed for python 3 compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9718) Remove pkg_resources.py from Impala/shell
[ https://issues.apache.org/jira/browse/IMPALA-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9718. - Fix Version/s: Impala 4.0 Resolution: Fixed > Remove pkg_resources.py from Impala/shell > - > > Key: IMPALA-9718 > URL: https://issues.apache.org/jira/browse/IMPALA-9718 > Project: IMPALA > Issue Type: Sub-task > Components: Clients >Affects Versions: Impala 4.0 >Reporter: David Knupp >Priority: Major > Fix For: Impala 4.0 > > > pkg_resources is available in the stdlib. There should be no need to bundle > it with the shell. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9721) Fix python 3 compatibility regression in impala-shell
[
https://issues.apache.org/jira/browse/IMPALA-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp resolved IMPALA-9721.
-
Resolution: Fixed
> Fix python 3 compatibility regression in impala-shell
> -
>
> Key: IMPALA-9721
> URL: https://issues.apache.org/jira/browse/IMPALA-9721
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Reporter: David Knupp
>Assignee: David Knupp
>Priority: Major
>
> The fix for IMPALA-9398 introduced a small regression with regard to python 3
> compatibility. We don't have python 3 tests yet to catch regression of this
> this type, and it was missed in code review.
> The regression happens in two places. An example is:
> https://github.com/apache/impala/blob/master/shell/impala_shell.py#L248
> The syntax for catching exceptions has changed in python 3 to require the
> "as" keyword.
> {noformat}
> try:
> do_stuff()
> except as e:
> panic()
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9667) TestImpalaShellInteractive failing as session not correctly closed
[
https://issues.apache.org/jira/browse/IMPALA-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099376#comment-17099376
]
David Knupp commented on IMPALA-9667:
-
So it seems like this issue (and IMPALA-9680) are regressions wholly caused by
https://gerrit.cloudera.org/c/15529/ ?
> TestImpalaShellInteractive failing as session not correctly closed
> --
>
> Key: IMPALA-9667
> URL: https://issues.apache.org/jira/browse/IMPALA-9667
> Project: IMPALA
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: David Knupp
>Priority: Critical
>
> TestImpalaShellInteractive.test_reconnect and
> TestImpalaShellInteractive.test_ddl_queries_are_closed fail waiting for
> metric value 'impala-server.num-open-beeswax-sessions' to be 0.
> The tests actually fail with
> {code}
> E TypeError: not all arguments converted during string formatting
> {code}
> which is the result of [IMPALA-9666]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9667) TestImpalaShellInteractive failing as session not correctly closed
[
https://issues.apache.org/jira/browse/IMPALA-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9667 started by David Knupp.
---
> TestImpalaShellInteractive failing as session not correctly closed
> --
>
> Key: IMPALA-9667
> URL: https://issues.apache.org/jira/browse/IMPALA-9667
> Project: IMPALA
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: David Knupp
>Priority: Critical
>
> TestImpalaShellInteractive.test_reconnect and
> TestImpalaShellInteractive.test_ddl_queries_are_closed fail waiting for
> metric value 'impala-server.num-open-beeswax-sessions' to be 0.
> The tests actually fail with
> {code}
> E TypeError: not all arguments converted during string formatting
> {code}
> which is the result of [IMPALA-9666]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9724) Setup python3 compile or else 'pylint --py3k' checking on jenkins.impala.io
David Knupp created IMPALA-9724:
---
Summary: Setup python3 compile or else 'pylint --py3k' checking on
jenkins.impala.io
Key: IMPALA-9724
URL: https://issues.apache.org/jira/browse/IMPALA-9724
Project: IMPALA
Issue Type: Sub-task
Affects Versions: Impala 4.0
Reporter: David Knupp
Until we get python3 testing integrated into the actual mini-cluster stack, we
should be able to add {{pylint --py3k}} check to the upstream build pipeline,
similar to how we used to check for python 2.6 compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9648) Exclude or update netty jar
[ https://issues.apache.org/jira/browse/IMPALA-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9648. - Resolution: Fixed > Exclude or update netty jar > --- > > Key: IMPALA-9648 > URL: https://issues.apache.org/jira/browse/IMPALA-9648 > Project: IMPALA > Issue Type: Task >Reporter: Abhishek Rawat >Assignee: David Knupp >Priority: Major > > Add exclusion for netty if not being used. Or update it to version 4.1.44 or > later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9721) Fix python 3 compatibility regression in impala-shell
[
https://issues.apache.org/jira/browse/IMPALA-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-9721:
Summary: Fix python 3 compatibility regression in impala-shell (was: Fix
python 3 compatibility regression)
> Fix python 3 compatibility regression in impala-shell
> -
>
> Key: IMPALA-9721
> URL: https://issues.apache.org/jira/browse/IMPALA-9721
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Reporter: David Knupp
>Assignee: David Knupp
>Priority: Major
>
> The fix for IMPALA-9398 introduced a small regression with regard to python 3
> compatibility. We don't have python 3 tests yet to catch regression of this
> this type, and it was missed in code review.
> The regression happens in two places. An example is:
> https://github.com/apache/impala/blob/master/shell/impala_shell.py#L248
> The syntax for catching exceptions has changed in python 3 to require the
> "as" keyword.
> {noformat}
> try:
> do_stuff()
> except as e:
> panic()
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9721) Fix python 3 compatibility regression
David Knupp created IMPALA-9721:
---
Summary: Fix python 3 compatibility regression
Key: IMPALA-9721
URL: https://issues.apache.org/jira/browse/IMPALA-9721
Project: IMPALA
Issue Type: Bug
Components: Clients
Reporter: David Knupp
Assignee: David Knupp
The fix for IMPALA-9398 introduced a small regression with regard to python 3
compatibility. We don't have python 3 tests yet to catch regression of this
this type, and it was missed in code review.
The regression happens in two places. An example is:
https://github.com/apache/impala/blob/master/shell/impala_shell.py#L248
The syntax for catching exceptions has changed in python 3 to require the "as"
keyword.
{noformat}
try:
do_stuff()
except as e:
panic()
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9649) Exclude shiro-crypto-core and shiro-core jars from maven download
[ https://issues.apache.org/jira/browse/IMPALA-9649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9649. - Resolution: Fixed > Exclude shiro-crypto-core and shiro-core jars from maven download > - > > Key: IMPALA-9649 > URL: https://issues.apache.org/jira/browse/IMPALA-9649 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > Fix For: Impala 4.0 > > > These jars have known security vulnerabilities. They are included as part of > Sentry, and are not used by Impala directly. > There's a currently a plan to remove Sentry altogether, but since will > require non-trivial effort, until that time, let's exclude these items from > the maven download. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9720) Upgrade bitarray from 0.9.0 to 1.2.1 in Impala/shell/ext-py
David Knupp created IMPALA-9720: --- Summary: Upgrade bitarray from 0.9.0 to 1.2.1 in Impala/shell/ext-py Key: IMPALA-9720 URL: https://issues.apache.org/jira/browse/IMPALA-9720 Project: IMPALA Issue Type: Sub-task Components: Clients Affects Versions: Impala 4.0 Reporter: David Knupp This is needed for python 3 compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9719) Upgrade sasl-0.1.1 to 0.2.1 in Impala/shell/ext-py
David Knupp created IMPALA-9719: --- Summary: Upgrade sasl-0.1.1 to 0.2.1 in Impala/shell/ext-py Key: IMPALA-9719 URL: https://issues.apache.org/jira/browse/IMPALA-9719 Project: IMPALA Issue Type: Sub-task Components: Clients Affects Versions: Impala 4.0 Reporter: David Knupp Needed for python 3 compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9718) Remove pkg_resources.py from Impala/shell
David Knupp created IMPALA-9718: --- Summary: Remove pkg_resources.py from Impala/shell Key: IMPALA-9718 URL: https://issues.apache.org/jira/browse/IMPALA-9718 Project: IMPALA Issue Type: Sub-task Components: Clients Affects Versions: Impala 4.0 Reporter: David Knupp pkg_resources is available in the stdlib. There should be no need to bundle it with the shell. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8608) Impala needs to support Python 3
[ https://issues.apache.org/jira/browse/IMPALA-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-8608. - Fix Version/s: Not Applicable Resolution: Duplicate > Impala needs to support Python 3 > > > Key: IMPALA-8608 > URL: https://issues.apache.org/jira/browse/IMPALA-8608 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Lars Volker >Priority: Critical > Fix For: Not Applicable > > > [The End of Python 2.7|https://pythonclock.org/] support is getting closer > and as such we need to be able to move to Python 3. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9667) TestImpalaShellInteractive failing as session not correctly closed
[
https://issues.apache.org/jira/browse/IMPALA-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096926#comment-17096926
]
David Knupp commented on IMPALA-9667:
-
[~tarmstrong] -- Thanks for the additional info.
> TestImpalaShellInteractive failing as session not correctly closed
> --
>
> Key: IMPALA-9667
> URL: https://issues.apache.org/jira/browse/IMPALA-9667
> Project: IMPALA
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: David Knupp
>Priority: Critical
>
> TestImpalaShellInteractive.test_reconnect and
> TestImpalaShellInteractive.test_ddl_queries_are_closed fail waiting for
> metric value 'impala-server.num-open-beeswax-sessions' to be 0.
> The tests actually fail with
> {code}
> E TypeError: not all arguments converted during string formatting
> {code}
> which is the result of [IMPALA-9666]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9648) Exclude or update netty jar
[ https://issues.apache.org/jira/browse/IMPALA-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9648. - Target Version: Impala 4.0 Resolution: Fixed > Exclude or update netty jar > --- > > Key: IMPALA-9648 > URL: https://issues.apache.org/jira/browse/IMPALA-9648 > Project: IMPALA > Issue Type: Task >Reporter: Abhishek Rawat >Assignee: David Knupp >Priority: Major > > Add exclusion for netty if not being used. Or update it to version 4.1.44 or > later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9647) Exclude or update fluent-hc jar
[ https://issues.apache.org/jira/browse/IMPALA-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9647. - Fix Version/s: Impala 4.0 Resolution: Fixed > Exclude or update fluent-hc jar > --- > > Key: IMPALA-9647 > URL: https://issues.apache.org/jira/browse/IMPALA-9647 > Project: IMPALA > Issue Type: Task >Reporter: Abhishek Rawat >Assignee: David Knupp >Priority: Blocker > Fix For: Impala 4.0 > > > Add exclusion for fluent-hc-4.3.2.jar or upgrade it to 4.3.6 or later version. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-9649) Exclude shiro-crypto-core and shiro-core jars from maven download
[ https://issues.apache.org/jira/browse/IMPALA-9649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp reopened IMPALA-9649: - > Exclude shiro-crypto-core and shiro-core jars from maven download > - > > Key: IMPALA-9649 > URL: https://issues.apache.org/jira/browse/IMPALA-9649 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > Fix For: Impala 4.0 > > > These jars have known security vulnerabilities. They are included as part of > Sentry, and are not used by Impala directly. > There's a currently a plan to remove Sentry altogether, but since will > require non-trivial effort, until that time, let's exclude these items from > the maven download. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9649) Exclude shiro-crypto-core and shiro-core jars from maven download
[ https://issues.apache.org/jira/browse/IMPALA-9649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091035#comment-17091035 ] David Knupp commented on IMPALA-9649: - Cc: [~dgarg] > Exclude shiro-crypto-core and shiro-core jars from maven download > - > > Key: IMPALA-9649 > URL: https://issues.apache.org/jira/browse/IMPALA-9649 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > Fix For: Impala 4.0 > > > These jars have known security vulnerabilities. They are included as part of > Sentry, and are not used by Impala directly. > There's a currently a plan to remove Sentry altogether, but since will > require non-trivial effort, until that time, let's exclude these items from > the maven download. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9648) Exclude or update netty jar
[ https://issues.apache.org/jira/browse/IMPALA-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089846#comment-17089846 ] David Knupp commented on IMPALA-9648: - Thanks [~tarmstrong] and [~tmate] -- the enforcer plugin was mentioned to me in another context, but I didn't grasp the full significance of it until now. I'll try that next. > Exclude or update netty jar > --- > > Key: IMPALA-9648 > URL: https://issues.apache.org/jira/browse/IMPALA-9648 > Project: IMPALA > Issue Type: Task >Reporter: Abhishek Rawat >Assignee: David Knupp >Priority: Major > > Add exclusion for netty if not being used. Or update it to version 4.1.44 or > later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9648) Exclude or update netty jar
[
https://issues.apache.org/jira/browse/IMPALA-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089187#comment-17089187
]
David Knupp commented on IMPALA-9648:
-
I'm having some trouble with syntax. Note that the netty dependency is now
appearing under Zookeeper, which is itself nested below hadoop where I
originally added the exclusion (org.apache.zookeeper isn't even explicitly
listed in our pom.xml, so I guess it's just pulled in automatically by hadoop).
If I try to add a zookeeper dependency node so that I can specify an exclusion
for netty (since all exclusions seem to be contained within dependencies,
otherwise you get syntax errors) I see:
{noformat}
[ERROR] 'dependencies.dependency.version' for
org.apache.zookeeper:zookeeper:jar is missing. @ line 79, column 17
{noformat}
Also, when I looked up excludes in the maven docs, it says this.
"Exclusions work on the entire dependency graph below the point where they are
declared. "
(from
https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html)
But this isn't the behavior I'm seeing. If I'm reading this correctly, since
zookeeper is nested below hadoop, I would expect that excluding netty from
hadoop would also mean it's excluded for all sub-dependencies as well.
I really don't know anything about maven, or how Java packages are built.
> Exclude or update netty jar
> ---
>
> Key: IMPALA-9648
> URL: https://issues.apache.org/jira/browse/IMPALA-9648
> Project: IMPALA
> Issue Type: Task
>Reporter: Abhishek Rawat
>Assignee: David Knupp
>Priority: Major
>
> Add exclusion for netty if not being used. Or update it to version 4.1.44 or
> later.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9648) Exclude or update netty jar
[ https://issues.apache.org/jira/browse/IMPALA-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089128#comment-17089128 ] David Knupp commented on IMPALA-9648: - So, would this be an indication that we can't get rid of this particular dependency? Should this item be handled by hadoop folks? > Exclude or update netty jar > --- > > Key: IMPALA-9648 > URL: https://issues.apache.org/jira/browse/IMPALA-9648 > Project: IMPALA > Issue Type: Task >Reporter: Abhishek Rawat >Assignee: David Knupp >Priority: Major > > Add exclusion for netty if not being used. Or update it to version 4.1.44 or > later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9648) Exclude or update netty jar
[
https://issues.apache.org/jira/browse/IMPALA-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089117#comment-17089117
]
David Knupp commented on IMPALA-9648:
-
[~joemcdonnell], [~tarmstr...@cloudera.com] or [~jfs] -- I wonder if you'd have
any insights here.
When I use mvn dependency:tree to look up netty on asf/master, I see this:
{noformat}
[INFO] org.apache.impala:impala-frontend:jar:0.1-SNAPSHOT
[...]
[INFO] +- org.apache.hadoop:hadoop-hdfs:jar:3.0.0-cdh6.x-SNAPSHOT:compile
[...]
[INFO] | +- io.netty:netty:jar:3.10.6.Final:compile
[INFO] | +- io.netty:netty-all:jar:4.1.42.Final:compile
{noformat}
So I added the following exclusion:
{noformat}
org.apache.hadoop
hadoop-hdfs
${hadoop.version}
org.eclipse.jetty
*
org.fusesource.leveldbjni
*
io.netty
*
{noformat}
But when I rebuild and check mvn dependency:tree again, I see new occurrences
of netty:
{noformat}
[INFO] org.apache.impala:impala-frontend:jar:0.1-SNAPSHOT
[...]
[INFO] +- org.apache.hadoop:hadoop-hdfs:jar:3.0.0-cdh6.x-SNAPSHOT:compile
[...]
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh6.x-SNAPSHOT:compile
[INFO] | | \- io.netty:netty:jar:3.10.6.Final:compile. <
this wasn't here before
[INFO] +- org.apache.hive:hive-hbase-handler:jar:2.1.1-cdh6.x-SNAPSHOT:compile
[INFO] | +- org.apache.hbase:hbase-server:jar:2.1.0-cdh6.x-SNAPSHOT:compile
[INFO] | | +- org.apache.hbase:hbase-http:jar:2.1.0-cdh6.x-SNAPSHOT:compile
[...]
[INFO] | | +-
org.apache.hadoop:hadoop-distcp:jar:3.0.0-cdh6.x-SNAPSHOT:compile
[INFO] | | | \- io.netty:netty-all:jar:4.1.42.Final:compile
< this wasn't here before
{noformat}
> Exclude or update netty jar
> ---
>
> Key: IMPALA-9648
> URL: https://issues.apache.org/jira/browse/IMPALA-9648
> Project: IMPALA
> Issue Type: Task
>Reporter: Abhishek Rawat
>Assignee: David Knupp
>Priority: Major
>
> Add exclusion for netty if not being used. Or update it to version 4.1.44 or
> later.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9648) Exclude or update netty jar
[ https://issues.apache.org/jira/browse/IMPALA-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9648 started by David Knupp. --- > Exclude or update netty jar > --- > > Key: IMPALA-9648 > URL: https://issues.apache.org/jira/browse/IMPALA-9648 > Project: IMPALA > Issue Type: Task >Reporter: Abhishek Rawat >Assignee: David Knupp >Priority: Major > > Add exclusion for netty if not being used. Or update it to version 4.1.44 or > later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9647) Exclude or update fluent-hc jar
[ https://issues.apache.org/jira/browse/IMPALA-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088000#comment-17088000 ] David Knupp commented on IMPALA-9647: - Patch submitted: http://gerrit.cloudera.org:8080/15760 > Exclude or update fluent-hc jar > --- > > Key: IMPALA-9647 > URL: https://issues.apache.org/jira/browse/IMPALA-9647 > Project: IMPALA > Issue Type: Task >Reporter: Abhishek Rawat >Assignee: David Knupp >Priority: Blocker > > Add exclusion for fluent-hc-4.3.2.jar or upgrade it to 4.3.6 or later version. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9647) Exclude or update fluent-hc jar
[ https://issues.apache.org/jira/browse/IMPALA-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9647 started by David Knupp. --- > Exclude or update fluent-hc jar > --- > > Key: IMPALA-9647 > URL: https://issues.apache.org/jira/browse/IMPALA-9647 > Project: IMPALA > Issue Type: Task >Reporter: Abhishek Rawat >Assignee: David Knupp >Priority: Blocker > > Add exclusion for fluent-hc-4.3.2.jar or upgrade it to 4.3.6 or later version. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
[
https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp resolved IMPALA-9489.
-
Fix Version/s: Impala 4.0
Resolution: Fixed
> Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
> --
>
> Key: IMPALA-9489
> URL: https://issues.apache.org/jira/browse/IMPALA-9489
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 3.4.0
>Reporter: David Knupp
>Assignee: David Knupp
>Priority: Major
> Fix For: Impala 4.0
>
>
> [Note: this JIRA was filed in relation to the ongoing effort to make the
> impala-shell compatible with python 3]
> The impala python development environment is a fairly convoluted affair -- a
> number of packages are installed in the infra/python/env, some of it comes
> from the toolchain, some of it is generated and lives in the shell directory.
> Generally speaking, if you launch impala-python and import a module, it's not
> necessarily easy to predict where the module might live.
> {noformat}
> $ python
> Python 2.7.10 (default, Aug 17 2018, 19:45:58)
> [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sasl
> >>> sasl
> '/home/systest/Impala/shell/ext-py/sasl-0.1.1/dist/sasl-0.1.1-py2.7-linux-x86_64.egg/sasl/__init__.pyc'>
> >>> import requests
> >>> requests
> '/home/systest/Impala/infra/python/env/local/lib/python2.7/site-packages/requests/__init__.pyc'>
> >>> import Logging
> >>> Logging
> '/home/systest/Impala/shell/gen-py/Logging/__init__.pyc'>
> >>> import thrift
> >>> thrift
> '/home/systest/Impala/toolchain/thrift-0.9.3-p7/python/lib/python2.7/site-packages/thrift/__init__.pyc'>
> {noformat}
> Really, there is no one coherent environment -- there's just whatever
> collection of modules happens to be available at a given time for a given
> type of invocation, all of which is accomplished behind the scenes by calling
> scripts like {{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}}
> that are responsible for cobbling together a PYTHONPATH based on known
> locations and current env variables.
> As far as I can tell, there are three important contexts where python comes
> into play...
> * during the build process (used during data load, e.g.,
> testdata/bin/load_nested.py)
> * when running the py.test bases e2e tests
> * whenever the impala-shell is invoked
> As noted by IMPALA-7825 (and also in a conversation I had with
> [~stakiar_impala_496e]), we're dependent on thrift 0.9.3 during the build
> process. This seems to come into play during the loading of test data
> (specifically, when calling testdata/bin/load_nested.py) mainly because at
> one point there was some well-intentioned but probably misguided attempt at
> code reuse from the test framework. The test code that gets re-used involves
> impyla and/or thrift-sasl, which currently still relies on thrift 0.9.3. So
> our test framework, and by extension the build, both inherit the same
> limitation.
> The impala-shell, on the other hand, luckily doesn't directly reuse any of
> the same test modules, and there really is no need to keep it pinned to
> 0.9.3. However, since calling the impala-shell.sh winds up invoking
> {{set-pythonpath.sh}}, the same script that script sets up the environment
> during building or testing, thrift 0.9.3 just kind of leaks over by default.
> As it turns out, thrift 0.9.3 is also one of the many limitations restricting
> the impala-shell to python 2. Luckily, with IMPALA-7924 resolved,
> thrift-0.11.0 is available -- we just have to use it. And the way to
> accomplish that is by decoupling the impala-shell from relying either
> {{set-pythonpath.sh}} or {{impala-python-common.sh}}.
> As a first pass, we can address the dev environment by just having
> {{impala-shell.sh}} itself do whatever is required to find python
> dependencies, and we can specify thrift-0.11.0 there. Also, thrift 0.11.0
> should be used by both of the scripts used to create the tarballs that
> package the impala-shell for customer environments. Neither of these should
> adversely building Impala or running the py.test test framework.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9501) Upgrade sqlparse to a version that supports python 3.0
[
https://issues.apache.org/jira/browse/IMPALA-9501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp resolved IMPALA-9501.
-
Fix Version/s: Impala 4.0
Resolution: Fixed
> Upgrade sqlparse to a version that supports python 3.0
> --
>
> Key: IMPALA-9501
> URL: https://issues.apache.org/jira/browse/IMPALA-9501
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Reporter: David Knupp
>Assignee: David Knupp
>Priority: Major
> Fix For: Impala 4.0
>
>
> The current version (0.1.19) was selected, per IMPALA-6999. because it's the
> last version to be compatible with python 2.6. However, it's not compatible
> with python 3.x.
> {noformat}
> Traceback (most recent call last):
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/impala_shell.py",
> line 37, in
> import sqlparse
> File "", line 983, in _find_and_load
> File "", line 967, in _find_and_load_unlocked
> File "", line 668, in _load_unlocked
> File "", line 638, in _load_backward_compatible
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/__init__.py",
> line 13, in
> File "", line 983, in _find_and_load
> File "", line 967, in _find_and_load_unlocked
> File "", line 668, in _load_unlocked
> File "", line 638, in _load_backward_compatible
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/engine/__init__.py",
> line 8, in
> File "", line 983, in _find_and_load
> File "", line 963, in _find_and_load_unlocked
> File "", line 906, in _find_spec
> File "", line 1280, in find_spec
> File "", line 1254, in _get_spec
> File "", line 1235, in
> _legacy_get_spec
> File "", line 441, in spec_from_loader
> File "", line 594, in
> spec_from_file_location
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/lexer.py",
> line 84
> except Exception, err:
> ^
> SyntaxError: invalid syntax
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9362) Update sqlparse used by impala-shell from version 0.1.19 to latest
[ https://issues.apache.org/jira/browse/IMPALA-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9362. - Fix Version/s: Impala 4.0 Resolution: Fixed > Update sqlparse used by impala-shell from version 0.1.19 to latest > -- > > Key: IMPALA-9362 > URL: https://issues.apache.org/jira/browse/IMPALA-9362 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > Fix For: Impala 4.0 > > > The fix for IMPALA-6337 involved correcting the way that sqlparse, an > upstream 3rd party python library used by the impala-shell, parses queries > that contain line breaks embedded inside of double quotes. Initially, > Impala's internally bundled version of sqlparse (based on 0.1.19) was > patched; meanwhile, a pull request to get the fix into an official release > was submitted. > That pull-request was finally included in the 0.3.0 version of sqlparse. > However, there were other changes to the library in the interim, in terms of > API's and also in some of the parsing logic, that breaks the impala-shell in > other ways, so simply migrating to the newer release is not straightforward. > We need to find and fix all the places that the newer sqlparse breaks the > impala-shell, so that we can stop relying on sqlparse 0.1.19 (which, in some > places, is not python 3 compatible). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9649) Exclude shiro-crypto-core and shiro-core jars from maven download
[ https://issues.apache.org/jira/browse/IMPALA-9649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9649. - Resolution: Fixed > Exclude shiro-crypto-core and shiro-core jars from maven download > - > > Key: IMPALA-9649 > URL: https://issues.apache.org/jira/browse/IMPALA-9649 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > Fix For: Impala 4.0 > > > These jars have known security vulnerabilities. They are included as part of > Sentry, and are not used by Impala directly. > There's a currently a plan to remove Sentry altogether, but since will > require non-trivial effort, until that time, let's exclude these items from > the maven download. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-9627) Update Impala utility Python scripts to be Python3 compatible
[ https://issues.apache.org/jira/browse/IMPALA-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084224#comment-17084224 ] David Knupp edited comment on IMPALA-9627 at 4/15/20, 4:29 PM: --- [~laszlog] -- Absolutely. As in the effort with IMPALA-3343, it's totally possible to make scripts compatible with both python 3.0 and 2.7+. was (Author: dknupp): Absolutely. As in the effort with IMPALA-3343, it's totally possible to make scripts compatible with both python 3.0 and 2.7+. > Update Impala utility Python scripts to be Python3 compatible > - > > Key: IMPALA-9627 > URL: https://issues.apache.org/jira/browse/IMPALA-9627 > Project: IMPALA > Issue Type: Improvement >Affects Versions: Impala 4.0 >Reporter: Laszlo Gaal >Priority: Major > > Impala utility Python scripts are the ones that run outside the Impala > virtualenv with system Python (not impala-python), except for impala-shell. > These scripts should be updated for Python3 to help the project's migration > to Python 3 after the retirement of Python 2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9627) Update Impala utility Python scripts to be Python3 compatible
[ https://issues.apache.org/jira/browse/IMPALA-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084224#comment-17084224 ] David Knupp commented on IMPALA-9627: - Absolutely. As in the effort with IMPALA-3343, it's totally possible to make scripts compatible with both python 3.0 and 2.7+. > Update Impala utility Python scripts to be Python3 compatible > - > > Key: IMPALA-9627 > URL: https://issues.apache.org/jira/browse/IMPALA-9627 > Project: IMPALA > Issue Type: Improvement >Affects Versions: Impala 4.0 >Reporter: Laszlo Gaal >Priority: Major > > Impala utility Python scripts are the ones that run outside the Impala > virtualenv with system Python (not impala-python), except for impala-shell. > These scripts should be updated for Python3 to help the project's migration > to Python 3 after the retirement of Python 2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9596) TestNestedTypesNoMtDop.test_tpch_mem_limit_single_node failed
[
https://issues.apache.org/jira/browse/IMPALA-9596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083387#comment-17083387
]
David Knupp commented on IMPALA-9596:
-
Seen again:
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2159/testReport/
> TestNestedTypesNoMtDop.test_tpch_mem_limit_single_node failed
> -
>
> Key: IMPALA-9596
> URL: https://issues.apache.org/jira/browse/IMPALA-9596
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 4.0
>Reporter: Yongzhi Chen
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, flaky
>
> parallel-all-tests-nightly failed with:
> query_test.test_nested_types.TestNestedTypesNoMtDop.test_tpch_mem_limit_single_node[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> orc/def/block] (from pytest)
> Error Message
> query_test/test_nested_types.py:159: in test_tpch_mem_limit_single_node
> new_vector, use_db='tpch_nested' + db_suffix)
> common/impala_test_suite.py:665: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:481: in __verify_exceptions (expected_str,
> actual_str) E AssertionError: Unexpected exception string. Expected:
> row_regex: .*Memory limit exceeded: Failed to allocate [0-9]+ bytes for
> collection 'tpch_nested_.*.customer.c_orders.item.o_lineitems'.* E Not
> found in actual: ImpalaBeeswaxException: Query aborted:Memory limit exceeded:
> Error occurred on backend a3c3b59bd11e:22000 by fragment
> c3440dc607fd4a18:25e42cf5Memory left in process limit: 8.26 GBMemory
> left in query limit: -3.94 KBQuery(c3440dc607fd4a18:25e42cf5): memory
> limit exceeded. Limit=20.00 MB Reservation=8.00 MB ReservationLimit=24.00 MB
> OtherMemory=12.00 MB Total=20.00 MB Peak=20.00 MB Fragment
> c3440dc607fd4a18:25e42cf5: Reservation=8.00 MB OtherMemory=12.00 MB
> Total=20.00 MB Peak=20.00 MBAGGREGATION_NODE (id=6): Total=24.00 KB
> Peak=24.00 KB NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
> Exprs: Total=4.00 KB Peak=4.00 KBSUBPLAN_NODE (id=1): Total=4.81 MB
> Peak=4.81 MBHDFS_SCAN_NODE (id=0): Reservation=8.00 MB OtherMemory=7.12
> MB Total=15.12 MB Peak=15.12 MBNESTED_LOOP_JOIN_NODE (id=5): Total=24.00
> KB Peak=24.00 KB Nested Loop Join Builder: Total=8.00 KB Peak=8.00 KB
> AGGREGATION_NODE (id=4): Total=24.00 KB Peak=24.00 KB
> NonGroupingAggregator 0: Total=16.00 KB Peak=16.00 KBExprs:
> Total=4.00 KB Peak=4.00 KBUNNEST_NODE (id=3): Total=0 Peak=0
> SINGULAR_ROW_SRC_NODE (id=2): Total=0 Peak=0PLAN_ROOT_SINK: Total=0
> Peak=0 CodeGen: Total=2.48 KB Peak=351.00 KB
> Stacktrace
> query_test/test_nested_types.py:159: in test_tpch_mem_limit_single_node
> new_vector, use_db='tpch_nested' + db_suffix)
> common/impala_test_suite.py:665: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:481: in __verify_exceptions
> (expected_str, actual_str)
> E AssertionError: Unexpected exception string. Expected: row_regex:
> .*Memory limit exceeded: Failed to allocate [0-9]+ bytes for collection
> 'tpch_nested_.*.customer.c_orders.item.o_lineitems'.*
> E Not found in actual: ImpalaBeeswaxException: Query aborted:Memory limit
> exceeded: Error occurred on backend a3c3b59bd11e:22000 by fragment
> c3440dc607fd4a18:25e42cf5Memory left in process limit: 8.26 GBMemory
> left in query limit: -3.94 KBQuery(c3440dc607fd4a18:25e42cf5): memory
> limit exceeded. Limit=20.00 MB Reservation=8.00 MB ReservationLimit=24.00 MB
> OtherMemory=12.00 MB Total=20.00 MB Peak=20.00 MB Fragment
> c3440dc607fd4a18:25e42cf5: Reservation=8.00 MB OtherMemory=12.00 MB
> Total=20.00 MB Peak=20.00 MBAGGREGATION_NODE (id=6): Total=24.00 KB
> Peak=24.00 KB NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
> Exprs: Total=4.00 KB Peak=4.00 KBSUBPLAN_NODE (id=1): Total=4.81 MB
> Peak=4.81 MBHDFS_SCAN_NODE (id=0): Reservation=8.00 MB OtherMemory=7.12
> MB Total=15.12 MB Peak=15.12 MBNESTED_LOOP_JOIN_NODE (id=5): Total=24.00
> KB Peak=24.00 KB Nested Loop Join Builder: Total=8.00 KB Peak=8.00 KB
> AGGREGATION_NODE (id=4): Total=24.00 KB Peak=24.00 KB
> NonGroupingAggregator 0: Total=16.00 KB Peak=16.00 KBExprs:
> Total=4.00 KB Peak=4.00 KBUNNEST_NODE (id=3): Total=0 Peak=0
> SINGULAR_ROW_SRC_NODE (id=2): Total=0 Peak=0PLAN_ROOT_SINK: Total=0
> Peak=0 CodeGen: Total=2.48 KB Peak=351.00 KB
[jira] [Work started] (IMPALA-9649) Exclude shiro-crypto-core and shiro-core jars from maven download
[ https://issues.apache.org/jira/browse/IMPALA-9649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9649 started by David Knupp. --- > Exclude shiro-crypto-core and shiro-core jars from maven download > - > > Key: IMPALA-9649 > URL: https://issues.apache.org/jira/browse/IMPALA-9649 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > Fix For: Impala 4.0 > > > These jars have known security vulnerabilities. They are included as part of > Sentry, and are not used by Impala directly. > There's a currently a plan to remove Sentry altogether, but since will > require non-trivial effort, until that time, let's exclude these items from > the maven download. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9649) Exclude shiro-crypto-core and shiro-core jars from maven download
[ https://issues.apache.org/jira/browse/IMPALA-9649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082681#comment-17082681 ] David Knupp commented on IMPALA-9649: - Patch submitted: https://gerrit.cloudera.org/c/15720/ > Exclude shiro-crypto-core and shiro-core jars from maven download > - > > Key: IMPALA-9649 > URL: https://issues.apache.org/jira/browse/IMPALA-9649 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > Fix For: Impala 4.0 > > > These jars have known security vulnerabilities. They are included as part of > Sentry, and are not used by Impala directly. > There's a currently a plan to remove Sentry altogether, but since will > require non-trivial effort, until that time, let's exclude these items from > the maven download. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9649) Exclude shiro-crypto-core and shiro-core jars from maven download
David Knupp created IMPALA-9649: --- Summary: Exclude shiro-crypto-core and shiro-core jars from maven download Key: IMPALA-9649 URL: https://issues.apache.org/jira/browse/IMPALA-9649 Project: IMPALA Issue Type: Bug Components: Frontend Affects Versions: Impala 4.0 Reporter: David Knupp Fix For: Impala 4.0 These jars have known security vulnerabilities. They are included as part of Sentry, and are not used by Impala directly. There's a currently a plan to remove Sentry altogether, but since will require non-trivial effort, until that time, let's exclude these items from the maven download. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9501) Upgrade sqlparse to a version that supports python 3.0
[
https://issues.apache.org/jira/browse/IMPALA-9501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076596#comment-17076596
]
David Knupp commented on IMPALA-9501:
-
cc: [~lenisha]
> Upgrade sqlparse to a version that supports python 3.0
> --
>
> Key: IMPALA-9501
> URL: https://issues.apache.org/jira/browse/IMPALA-9501
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Reporter: David Knupp
>Assignee: David Knupp
>Priority: Major
>
> The current version (0.1.19) was selected, per IMPALA-6999. because it's the
> last version to be compatible with python 2.6. However, it's not compatible
> with python 3.x.
> {noformat}
> Traceback (most recent call last):
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/impala_shell.py",
> line 37, in
> import sqlparse
> File "", line 983, in _find_and_load
> File "", line 967, in _find_and_load_unlocked
> File "", line 668, in _load_unlocked
> File "", line 638, in _load_backward_compatible
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/__init__.py",
> line 13, in
> File "", line 983, in _find_and_load
> File "", line 967, in _find_and_load_unlocked
> File "", line 668, in _load_unlocked
> File "", line 638, in _load_backward_compatible
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/engine/__init__.py",
> line 8, in
> File "", line 983, in _find_and_load
> File "", line 963, in _find_and_load_unlocked
> File "", line 906, in _find_spec
> File "", line 1280, in find_spec
> File "", line 1254, in _get_spec
> File "", line 1235, in
> _legacy_get_spec
> File "", line 441, in spec_from_loader
> File "", line 594, in
> spec_from_file_location
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/lexer.py",
> line 84
> except Exception, err:
> ^
> SyntaxError: invalid syntax
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9362) Update sqlparse used by impala-shell from version 0.1.19 to latest
[ https://issues.apache.org/jira/browse/IMPALA-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076594#comment-17076594 ] David Knupp commented on IMPALA-9362: - cc: [~lenisha] > Update sqlparse used by impala-shell from version 0.1.19 to latest > -- > > Key: IMPALA-9362 > URL: https://issues.apache.org/jira/browse/IMPALA-9362 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > The fix for IMPALA-6337 involved correcting the way that sqlparse, an > upstream 3rd party python library used by the impala-shell, parses queries > that contain line breaks embedded inside of double quotes. Initially, > Impala's internally bundled version of sqlparse (based on 0.1.19) was > patched; meanwhile, a pull request to get the fix into an official release > was submitted. > That pull-request was finally included in the 0.3.0 version of sqlparse. > However, there were other changes to the library in the interim, in terms of > API's and also in some of the parsing logic, that breaks the impala-shell in > other ways, so simply migrating to the newer release is not straightforward. > We need to find and fix all the places that the newer sqlparse breaks the > impala-shell, so that we can stop relying on sqlparse 0.1.19 (which, in some > places, is not python 3 compatible). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9501) Upgrade sqlparse to a version that supports python 3.0
[
https://issues.apache.org/jira/browse/IMPALA-9501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9501 started by David Knupp.
---
> Upgrade sqlparse to a version that supports python 3.0
> --
>
> Key: IMPALA-9501
> URL: https://issues.apache.org/jira/browse/IMPALA-9501
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Reporter: David Knupp
>Assignee: David Knupp
>Priority: Major
>
> The current version (0.1.19) was selected, per IMPALA-6999. because it's the
> last version to be compatible with python 2.6. However, it's not compatible
> with python 3.x.
> {noformat}
> Traceback (most recent call last):
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/impala_shell.py",
> line 37, in
> import sqlparse
> File "", line 983, in _find_and_load
> File "", line 967, in _find_and_load_unlocked
> File "", line 668, in _load_unlocked
> File "", line 638, in _load_backward_compatible
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/__init__.py",
> line 13, in
> File "", line 983, in _find_and_load
> File "", line 967, in _find_and_load_unlocked
> File "", line 668, in _load_unlocked
> File "", line 638, in _load_backward_compatible
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/engine/__init__.py",
> line 8, in
> File "", line 983, in _find_and_load
> File "", line 963, in _find_and_load_unlocked
> File "", line 906, in _find_spec
> File "", line 1280, in find_spec
> File "", line 1254, in _get_spec
> File "", line 1235, in
> _legacy_get_spec
> File "", line 441, in spec_from_loader
> File "", line 594, in
> spec_from_file_location
> File
> "/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/lexer.py",
> line 84
> except Exception, err:
> ^
> SyntaxError: invalid syntax
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9362) Update sqlparse used by impala-shell from version 0.1.19 to latest
[ https://issues.apache.org/jira/browse/IMPALA-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9362 started by David Knupp. --- > Update sqlparse used by impala-shell from version 0.1.19 to latest > -- > > Key: IMPALA-9362 > URL: https://issues.apache.org/jira/browse/IMPALA-9362 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > The fix for IMPALA-6337 involved correcting the way that sqlparse, an > upstream 3rd party python library used by the impala-shell, parses queries > that contain line breaks embedded inside of double quotes. Initially, > Impala's internally bundled version of sqlparse (based on 0.1.19) was > patched; meanwhile, a pull request to get the fix into an official release > was submitted. > That pull-request was finally included in the 0.3.0 version of sqlparse. > However, there were other changes to the library in the interim, in terms of > API's and also in some of the parsing logic, that breaks the impala-shell in > other ways, so simply migrating to the newer release is not straightforward. > We need to find and fix all the places that the newer sqlparse breaks the > impala-shell, so that we can stop relying on sqlparse 0.1.19 (which, in some > places, is not python 3 compatible). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9582) Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell
[ https://issues.apache.org/jira/browse/IMPALA-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072452#comment-17072452 ] David Knupp commented on IMPALA-9582: - https://gerrit.cloudera.org/c/15610/ > Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell > -- > > Key: IMPALA-9582 > URL: https://issues.apache.org/jira/browse/IMPALA-9582 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Blocker > Fix For: Impala 3.4.0 > > > thrift_sasl 0.4.1 introduced a regression whereby the Thrift transport was > not reading all data, causing clients to hang. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9582) Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell
[ https://issues.apache.org/jira/browse/IMPALA-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9582. - Resolution: Fixed > Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell > -- > > Key: IMPALA-9582 > URL: https://issues.apache.org/jira/browse/IMPALA-9582 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Blocker > Fix For: Impala 3.4.0 > > > thrift_sasl 0.4.1 introduced a regression whereby the Thrift transport was > not reading all data, causing clients to hang. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9582) Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell
[ https://issues.apache.org/jira/browse/IMPALA-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp updated IMPALA-9582: Priority: Blocker (was: Critical) > Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell > -- > > Key: IMPALA-9582 > URL: https://issues.apache.org/jira/browse/IMPALA-9582 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Blocker > Fix For: Impala 3.4.0 > > > thrift_sasl 0.4.1 introduced a regression whereby the Thrift transport was > not reading all data, causing clients to hang. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9582) Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell
[ https://issues.apache.org/jira/browse/IMPALA-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp updated IMPALA-9582: Issue Type: Bug (was: Improvement) > Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell > -- > > Key: IMPALA-9582 > URL: https://issues.apache.org/jira/browse/IMPALA-9582 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Critical > Fix For: Impala 3.4.0 > > > thrift_sasl 0.4.1 introduced a regression whereby the Thrift transport was > not reading all data, causing clients to hang. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9582) Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell
David Knupp created IMPALA-9582: --- Summary: Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell Key: IMPALA-9582 URL: https://issues.apache.org/jira/browse/IMPALA-9582 Project: IMPALA Issue Type: Improvement Components: Clients Affects Versions: Impala 3.4.0 Reporter: David Knupp Fix For: Impala 3.4.0 thrift_sasl 0.4.1 introduced a regression whereby the Thrift transport was not reading all data, causing clients to hang. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9582) Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell
[ https://issues.apache.org/jira/browse/IMPALA-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp reassigned IMPALA-9582: --- Assignee: David Knupp > Update thift_sasl 0.4.1 --> 0.4.2 for impala-shell > -- > > Key: IMPALA-9582 > URL: https://issues.apache.org/jira/browse/IMPALA-9582 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Critical > Fix For: Impala 3.4.0 > > > thrift_sasl 0.4.1 introduced a regression whereby the Thrift transport was > not reading all data, causing clients to hang. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Issue Comment Deleted] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
[
https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-9489:
Comment: was deleted
(was: Review available at https://gerrit.cloudera.org/c/15417/)
> Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
> --
>
> Key: IMPALA-9489
> URL: https://issues.apache.org/jira/browse/IMPALA-9489
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 3.4.0
>Reporter: David Knupp
>Assignee: David Knupp
>Priority: Major
>
> [Note: this JIRA was filed in relation to the ongoing effort to make the
> impala-shell compatible with python 3]
> The impala python development environment is a fairly convoluted affair -- a
> number of packages are installed in the infra/python/env, some of it comes
> from the toolchain, some of it is generated and lives in the shell directory.
> Generally speaking, if you launch impala-python and import a module, it's not
> necessarily easy to predict where the module might live.
> {noformat}
> $ python
> Python 2.7.10 (default, Aug 17 2018, 19:45:58)
> [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sasl
> >>> sasl
> '/home/systest/Impala/shell/ext-py/sasl-0.1.1/dist/sasl-0.1.1-py2.7-linux-x86_64.egg/sasl/__init__.pyc'>
> >>> import requests
> >>> requests
> '/home/systest/Impala/infra/python/env/local/lib/python2.7/site-packages/requests/__init__.pyc'>
> >>> import Logging
> >>> Logging
> '/home/systest/Impala/shell/gen-py/Logging/__init__.pyc'>
> >>> import thrift
> >>> thrift
> '/home/systest/Impala/toolchain/thrift-0.9.3-p7/python/lib/python2.7/site-packages/thrift/__init__.pyc'>
> {noformat}
> Really, there is no one coherent environment -- there's just whatever
> collection of modules happens to be available at a given time for a given
> type of invocation, all of which is accomplished behind the scenes by calling
> scripts like {{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}}
> that are responsible for cobbling together a PYTHONPATH based on known
> locations and current env variables.
> As far as I can tell, there are three important contexts where python comes
> into play...
> * during the build process (used during data load, e.g.,
> testdata/bin/load_nested.py)
> * when running the py.test bases e2e tests
> * whenever the impala-shell is invoked
> As noted by IMPALA-7825 (and also in a conversation I had with
> [~stakiar_impala_496e]), we're dependent on thrift 0.9.3 during the build
> process. This seems to come into play during the loading of test data
> (specifically, when calling testdata/bin/load_nested.py) mainly because at
> one point there was some well-intentioned but probably misguided attempt at
> code reuse from the test framework. The test code that gets re-used involves
> impyla and/or thrift-sasl, which currently still relies on thrift 0.9.3. So
> our test framework, and by extension the build, both inherit the same
> limitation.
> The impala-shell, on the other hand, luckily doesn't directly reuse any of
> the same test modules, and there really is no need to keep it pinned to
> 0.9.3. However, since calling the impala-shell.sh winds up invoking
> {{set-pythonpath.sh}}, the same script that script sets up the environment
> during building or testing, thrift 0.9.3 just kind of leaks over by default.
> As it turns out, thrift 0.9.3 is also one of the many limitations restricting
> the impala-shell to python 2. Luckily, with IMPALA-7924 resolved,
> thrift-0.11.0 is available -- we just have to use it. And the way to
> accomplish that is by decoupling the impala-shell from relying either
> {{set-pythonpath.sh}} or {{impala-python-common.sh}}.
> As a first pass, we can address the dev environment by just having
> {{impala-shell.sh}} itself do whatever is required to find python
> dependencies, and we can specify thrift-0.11.0 there. Also, thrift 0.11.0
> should be used by both of the scripts used to create the tarballs that
> package the impala-shell for customer environments. Neither of these should
> adversely building Impala or running the py.test test framework.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8908) Bad error message when failing to connect to HTTPS endpoint with shell
[
https://issues.apache.org/jira/browse/IMPALA-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061950#comment-17061950
]
David Knupp commented on IMPALA-8908:
-
[~tmate]-- sorry, didn't see this comment for some reason.
Yes, I suspect that resolving IMPALA-9489 will resolve this issue.
> Bad error message when failing to connect to HTTPS endpoint with shell
> --
>
> Key: IMPALA-8908
> URL: https://issues.apache.org/jira/browse/IMPALA-8908
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Affects Versions: Impala 3.3.0
>Reporter: Tim Armstrong
>Assignee: Tamas Mate
>Priority: Critical
> Labels: observability, ramp-up
>
> Legitimate connection errors get masked with an UnboundLocalError. It looks
> like THRIFT-3634 fixed this.
> {noformat}
> $ impala-shell.sh -i ip-10-97-80-186.cloudera.site --protocol=hs2 --ldap
> --user csso_tarmstrong --ssl
> Starting Impala Shell using LDAP-based authentication
> SSL is enabled. Impala server certificates will NOT be verified (set
> --ca_cert to change)
> LDAP password for csso_tarmstrong:
> Traceback (most recent call last):
> File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line
> 1880, in
> impala_shell_main()
> File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line
> 1841, in impala_shell_main
> with ImpalaShell(options, query_options) as shell:
> File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line
> 243, in __init__
> self.do_connect(options.impalad)
> File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line
> 812, in do_connect
> self._connect()
> File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line
> 860, in _connect
> self.server_version, self.webserver_address = self.imp_client.connect()
> File "/home/tarmstrong/Impala/incubator-impala/shell/impala_client.py",
> line 176, in connect
> self.transport = self._get_transport(self.client_connect_timeout_ms)
> File "/home/tarmstrong/Impala/incubator-impala/shell/impala_client.py",
> line 472, in _get_transport
> transport.open()
> File "/home/tarmstrong/Impala/incubator-impala/shell/thrift_sasl.py", line
> 61, in open
> self._trans.open()
> File
> "/opt/Impala-Toolchain/thrift-0.9.3-p7/python/lib/python2.7/site-packages/thrift/transport/TSSLSocket.py",
> line 258, in open
> logger.error('Error while connecting with %s.', ip_port, exc_info=True)
> UnboundLocalError: local variable 'ip_port' referenced before assignment
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9501) Upgrade sqlparse to a version that supports python 3.0
David Knupp created IMPALA-9501:
---
Summary: Upgrade sqlparse to a version that supports python 3.0
Key: IMPALA-9501
URL: https://issues.apache.org/jira/browse/IMPALA-9501
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Reporter: David Knupp
The current version (0.1.19) was selected, per IMPALA-6999. because it's the
last version to be compatible with python 2.6. However, it's not compatible
with python 3.x.
{noformat}
Traceback (most recent call last):
File
"/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/impala_shell.py",
line 37, in
import sqlparse
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 668, in _load_unlocked
File "", line 638, in _load_backward_compatible
File
"/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/__init__.py",
line 13, in
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 668, in _load_unlocked
File "", line 638, in _load_backward_compatible
File
"/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/engine/__init__.py",
line 8, in
File "", line 983, in _find_and_load
File "", line 963, in _find_and_load_unlocked
File "", line 906, in _find_spec
File "", line 1280, in find_spec
File "", line 1254, in _get_spec
File "", line 1235, in _legacy_get_spec
File "", line 441, in spec_from_loader
File "", line 594, in
spec_from_file_location
File
"/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/ext-py/sqlparse-0.1.19-py2.7.egg/sqlparse/lexer.py",
line 84
except Exception, err:
^
SyntaxError: invalid syntax
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
[
https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-9489:
Description:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of it is generated and lives in the shell directory.
Generally speaking, if you launch impala-python and import a module, it's not
necessarily easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just whatever
collection of modules happens to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by calling
scripts like {{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that
are responsible for cobbling together a PYTHONPATH based on known locations and
current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a conversation I had with
[~stakiar_impala_496e]), we're dependent on thrift 0.9.3 during the build
process. This seems to come into play during the loading of test data
(specifically, when calling testdata/bin/load_nested.py) mainly because at one
point there was some well-intentioned but probably misguided attempt at code
reuse from the test framework. The test code that gets re-used involves impyla
and/or thrift-sasl, which currently still relies on thrift 0.9.3. So our test
framework, and by extension the build, both inherit the same limitation.
The impala-shell, on the other hand, luckily doesn't directly reuse any of the
same test modules, and there really is no need to keep it pinned to 0.9.3.
However, since calling the impala-shell.sh winds up invoking
{{set-pythonpath.sh}}, the same script that script sets up the environment
during building or testing, thrift 0.9.3 just kind of leaks over by default.
As it turns out, thrift 0.9.3 is also one of the many limitations restricting
the impala-shell to python 2. Luckily, with IMPALA-7924 resolved, thrift-0.11.0
is available -- we just have to use it. And the way to accomplish that is by
decoupling the impala-shell from relying either {{set-pythonpath.sh}} or
{{impala-python-common.sh}}.
As a first pass, we can address the dev environment by just having
{{impala-shell.sh}} itself do whatever is required to find python dependencies,
and we can specify thrift-0.11.0 there. Also, thrift 0.11.0 should be used by
both of the scripts used to create the tarballs that package the impala-shell
for customer environments. Neither of these should adversely building Impala or
running the py.test test framework.
was:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of it is generated and lives in the shell directory.
Generally speaking, if you launch impala-python and import a module, it's not
necessarily easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just whatever
collection of modules happens to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by calling
scripts like {{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that
are responsible for cobbling together a PYTHONPATH based on known locations and
current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a
[jira] [Updated] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
[
https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-9489:
Description:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of it is generated and lives in the shell directory.
Generally speaking, if you launch impala-python and import a module, it's not
necessarily easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just whatever
collection of modules happens to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by calling
scripts like {{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that
are responsible for cobbling together a PYTHONPATH based on known locations and
current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a conversation I had with
[~stakiar_impala_496e]), we're dependent on thrift 0.9.3 during the build
process. This seems to come into play during the loading of test data
(specifically, when calling testdata/bin/load_nested.py) mainly because at one
point there was some well-intentioned but probably misguided attempt at code
reuse from the test framework. The test code that gets re-used involves impyla
and/or thrift-sasl, which currently still relies on thrift 0.9.3. So our test
framework, and by extension the build, both inherit the same limitation.
The impala-shell, on the other hand, luckily doesn't directly reuse any of the
same test modules, and there really is no need to keep it pinned to 0.9.3.
However, since calling the impala-shell.sh winds up invoking
{{set-pythonpath.sh}}, the same script that script sets up the environment
during building or testing, thrift 0.9.3 just kind of leaks over by default.
As it turns out, thrift 0.9.3 is also one of the many limitations restricting
the impala-shell to python 2. Luckily, with IMPALA-7924 resolved, thrift-0.11.0
is available -- we just have to use it. And the way to accomplish that is by
decoupling the impala-shell from relying either {{set-pythonpath.sh}} or
{{impala-python-common.sh}}.
was:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of it is generated and lives in the shell directory.
Generally speaking, if you launch impala-python and import a module, it's not
necessarily easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just whatever
collection of modules happens to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by calling
scripts like {{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that
are responsible for cobbling together a PYTHONPATH based on known locations and
current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a conversation I had with
[~stakiar_impala_496e]), we're dependent on thrift 0.9.3 the build process. It
seems to happen during test data load (specifically, when calling
testdata/bin/load_nested.py) mainly because there was some well-intentioned but
probably misjudged attempt at code reuse from the test framework. The test code
that gets re-used involves impyla and/or thrift-sasl, which currently still
relies on thrift
[jira] [Updated] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
[
https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-9489:
Description:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of it is generated and lives in the shell directory.
Generally speaking, if you launch impala-python and import a module, it's not
necessarily easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just whatever
collection of modules happens to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by scripts like
{{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that cobble
together a PYTHONPATH based on known locations and current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a conversation I had with
[~stakiar_impala_496e]), we're dependent on thrift 0.9.3 the build process. It
seems to happen during test data load (specifically, when calling
testdata/bin/load_nested.py) mainly because there was some well-intentioned but
probably misjudged attempt at code reuse from the test framework. The test code
that gets re-used involves impyla and/or thrift-sasl, which currently still
relies on thrift 0.9.3. So our test framework, and by extension the build, both
inherit the same limitation.
The impala-shell, on the other hand, luckily doesn't directly reuse any of the
same modules, and there's no real need to keep it pinned to 0.9.3. However,
since calling the impala-shell.sh winds up invoking {{set-pythonpath.sh}}, the
same script that script sets up the environment during building or testing, the
shell winds up defaulting to thrift 0.9.3 as well.
thrift 0.9.3 is one of the many limitations restricting the impala-shell to
python 2. Luckily, with IMPALA-7924 resolved, thrift-0.11.0 is available -- we
just have to use it. The way to accomplish this is be decoupling the
impala-shell from calling either {{set-pythonpath.sh}} or
{{impala-python-common.sh}}.
was:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of it is generated and lives in the shell directory.
Generally speaking, if you launch impala-python and import a module, it's not
necessarily easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just the collection of
available modules that happen to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by scripts like
{{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that cobble
together a PYTHONPATH based on known locations and current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a conversation I had with
[~stakiar_impala_496e]), we're dependent on thrift 0.9.3 the build process. It
seems to happen during test data load (specifically, when calling
testdata/bin/load_nested.py) mainly because there was some well-intentioned but
probably misjudged attempt at code reuse from the test framework. The test code
that gets re-used involves impyla and/or thrift-sasl, which currently still
relies on thrift 0.9.3. So our test framework, and by extension the build, both
inherit the same limitation.
The impala-shell, on the other
[jira] [Updated] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
[
https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-9489:
Description:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of it is generated and lives in the shell directory.
Generally speaking, if you launch impala-python and import a module, it's not
necessarily easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just whatever
collection of modules happens to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by calling
scripts like {{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that
are responsible for cobbling together a PYTHONPATH based on known locations and
current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a conversation I had with
[~stakiar_impala_496e]), we're dependent on thrift 0.9.3 the build process. It
seems to happen during test data load (specifically, when calling
testdata/bin/load_nested.py) mainly because there was some well-intentioned but
probably misjudged attempt at code reuse from the test framework. The test code
that gets re-used involves impyla and/or thrift-sasl, which currently still
relies on thrift 0.9.3. So our test framework, and by extension the build, both
inherit the same limitation.
The impala-shell, on the other hand, luckily doesn't directly reuse any of the
same modules, and there's no real need to keep it pinned to 0.9.3. However,
since calling the impala-shell.sh winds up invoking {{set-pythonpath.sh}}, the
same script that script sets up the environment during building or testing, the
shell winds up defaulting to thrift 0.9.3 as well.
thrift 0.9.3 is one of the many limitations restricting the impala-shell to
python 2. Luckily, with IMPALA-7924 resolved, thrift-0.11.0 is available -- we
just have to use it. The way to accomplish this is be decoupling the
impala-shell from calling either {{set-pythonpath.sh}} or
{{impala-python-common.sh}}.
was:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of it is generated and lives in the shell directory.
Generally speaking, if you launch impala-python and import a module, it's not
necessarily easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just whatever
collection of modules happens to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by scripts like
{{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that cobble
together a PYTHONPATH based on known locations and current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a conversation I had with
[~stakiar_impala_496e]), we're dependent on thrift 0.9.3 the build process. It
seems to happen during test data load (specifically, when calling
testdata/bin/load_nested.py) mainly because there was some well-intentioned but
probably misjudged attempt at code reuse from the test framework. The test code
that gets re-used involves impyla and/or thrift-sasl, which currently still
relies on thrift 0.9.3. So our test framework, and by extension the build, both
inherit the same limitation.
The
[jira] [Updated] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
[
https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-9489:
Description:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of it is generated and lives in the shell directory.
Generally speaking, if you launch impala-python and import a module, it's not
necessarily easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just the collection of
available modules that happen to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by scripts like
{{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that cobble
together a PYTHONPATH based on known locations and current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a conversation I had with
[~stakiar_impala_496e]), we're dependent on thrift 0.9.3 the build process. It
seems to happen during test data load (specifically, when calling
testdata/bin/load_nested.py) mainly because there was some well-intentioned but
probably misjudged attempt at code reuse from the test framework. The test code
that gets re-used involves impyla and/or thrift-sasl, which currently still
relies on thrift 0.9.3. So our test framework, and by extension the build, both
inherit the same limitation.
The impala-shell, on the other hand, luckily doesn't directly reuse any of the
same modules, and there's no real need to keep it pinned to 0.9.3. However,
since calling the impala-shell.sh winds up invoking {{set-pythonpath.sh}}, the
same script that script sets up the environment during building or testing, the
shell winds up defaulting to thrift 0.9.3 as well.
thrift 0.9.3 is one of the many limitations restricting the impala-shell to
python 2. Luckily, with IMPALA-7924 resolved, thrift-0.11.0 is available -- we
just have to use it. The way to accomplish this is be decoupling the
impala-shell from calling either {{set-pythonpath.sh}} or
{{impala-python-common.sh}}.
was:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of is generated and lives in the shell directory. Generally
speaking, if you launch impala-python and import a module, it's not necessarily
easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just the collection of
available modules that happen to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by scripts like
{{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that cobble
together a PYTHONPATH based on known locations and current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a conversation I had with
[~stakiar_impala_496e]), we're dependent on thrift 0.9.3 the build process. It
seems to happen during test data load (specifically, when calling
testdata/bin/load_nested.py) mainly because there was some well-intentioned but
probably misjudged attempt at code reuse from the test framework. The test code
that gets re-used involves impyla and/or thrift-sasl, which currently still
relies on thrift 0.9.3. So our test framework, and by extension the build, both
inherit the same limitation.
The impala-shell, on the
[jira] [Updated] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
[
https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-9489:
Description:
[Note: this JIRA was filed in relation to the ongoing effort to make the
impala-shell compatible with python 3]
The impala python development environment is a fairly convoluted affair -- a
number of packages are installed in the infra/python/env, some of it comes from
the toolchain, some of is generated and lives in the shell directory. Generally
speaking, if you launch impala-python and import a module, it's not necessarily
easy to predict where the module might live.
{noformat}
$ python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sasl
>>> sasl
>>> import requests
>>> requests
>>> import Logging
>>> Logging
>>> import thrift
>>> thrift
{noformat}
Really, there is no one coherent environment -- there's just the collection of
available modules that happen to be available at a given time for a given type
of invocation, all of which is accomplished behind the scenes by scripts like
{{bin/set-pythonpath.sh}} and {{bin/impala-python-common.sh}} that cobble
together a PYTHONPATH based on known locations and current env variables.
As far as I can tell, there are three important contexts where python comes
into play...
* during the build process (used during data load, e.g.,
testdata/bin/load_nested.py)
* when running the py.test bases e2e tests
* whenever the impala-shell is invoked
As noted by IMPALA-7825 (and also in a conversation I had with
[~stakiar_impala_496e]), we're dependent on thrift 0.9.3 the build process. It
seems to happen during test data load (specifically, when calling
testdata/bin/load_nested.py) mainly because there was some well-intentioned but
probably misjudged attempt at code reuse from the test framework. The test code
that gets re-used involves impyla and/or thrift-sasl, which currently still
relies on thrift 0.9.3. So our test framework, and by extension the build, both
inherit the same limitation.
The impala-shell, on the other hand, luckily doesn't directly reuse any of the
same modules, and there's no real need to keep it pinned to 0.9.3. However,
since calling the impala-shell.sh winds up invoking {{set-pythonpath.sh}}, the
same script that script sets up the environment during building or testing, the
shell winds up defaulting to thrift 0.9.3 as well.
thrift 0.9.3 is one of the many limitations restricting the impala-shell to
python 2. Luckily, with IMPALA-7924 resolved, thrift-0.11.0 is available -- we
just have to use it. The way to accomplish this is be decoupling the
impala-shell from calling either {{set-pythonpath.sh}} or
{{impala-python-common.sh}}.
was:
The impala python development environment is a fairly convoluted affair. A
number of packages are
Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build needs some
attention before that can happen, per IMPALA-7825, and also conversation I had
with [~stakiar_impala_496e].
However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python
files, and we should use those by default. It turns out that being stuck with
thrift-0.9.3 is a major impediment to achieving python 3 compatibility for our
python stack.
> Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
> --
>
> Key: IMPALA-9489
> URL: https://issues.apache.org/jira/browse/IMPALA-9489
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 3.4.0
>Reporter: David Knupp
>Assignee: David Knupp
>Priority: Major
>
> [Note: this JIRA was filed in relation to the ongoing effort to make the
> impala-shell compatible with python 3]
> The impala python development environment is a fairly convoluted affair -- a
> number of packages are installed in the infra/python/env, some of it comes
> from the toolchain, some of is generated and lives in the shell directory.
> Generally speaking, if you launch impala-python and import a module, it's not
> necessarily easy to predict where the module might live.
> {noformat}
> $ python
> Python 2.7.10 (default, Aug 17 2018, 19:45:58)
> [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sasl
> >>> sasl
> '/home/systest/Impala/shell/ext-py/sasl-0.1.1/dist/sasl-0.1.1-py2.7-linux-x86_64.egg/sasl/__init__.pyc'>
> >>> import requests
> >>> requests
> '/home/systest/Impala/infra/python/env/local/lib/python2.7/site-packages/requests/__init__.pyc'>
> >>> import Logging
> >>> Logging
[jira] [Updated] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
[ https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp updated IMPALA-9489: Description: The impala python development environment is a fairly convoluted affair. A number of packages are Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build needs some attention before that can happen, per IMPALA-7825, and also conversation I had with [~stakiar_impala_496e]. However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python files, and we should use those by default. It turns out that being stuck with thrift-0.9.3 is a major impediment to achieving python 3 compatibility for our python stack. was: Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build needs some attention before that can happen, per IMPALA-7825, and also conversation I had with [~stakiar_impala_496e]. However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python files, and we should use those by default. It turns out that being stuck with thrift-0.9.3 is a major impediment to achieving python 3 compatibility for our python stack. > Setup impala-shell.sh env separately, and use thrift-0.11.0 by default > -- > > Key: IMPALA-9489 > URL: https://issues.apache.org/jira/browse/IMPALA-9489 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > The impala python development environment is a fairly convoluted affair. A > number of packages are > Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build needs > some attention before that can happen, per IMPALA-7825, and also conversation > I had with [~stakiar_impala_496e]. > However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python > files, and we should use those by default. It turns out that being stuck with > thrift-0.9.3 is a major impediment to achieving python 3 compatibility for > our python stack. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default
[ https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp updated IMPALA-9489: Summary: Setup impala-shell.sh env separately, and use thrift-0.11.0 by default (was: Setup impala-shell.sh env separately, and use thrift-0.11.0 by default.) > Setup impala-shell.sh env separately, and use thrift-0.11.0 by default > -- > > Key: IMPALA-9489 > URL: https://issues.apache.org/jira/browse/IMPALA-9489 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build needs > some attention before that can happen, per IMPALA-7825, and also conversation > I had with [~stakiar_impala_496e]. > However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python > files, and we should use those by default. It turns out that being stuck with > thrift-0.9.3 is a major impediment to achieving python 3 compatibility for > our python stack. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9489) Setup impala-shell.sh env separately, and use thrift-0.11.0 by default.
[ https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp updated IMPALA-9489: Summary: Setup impala-shell.sh env separately, and use thrift-0.11.0 by default. (was: Have impala-python env and impala-shell use available thrift-0.11.0 files) > Setup impala-shell.sh env separately, and use thrift-0.11.0 by default. > --- > > Key: IMPALA-9489 > URL: https://issues.apache.org/jira/browse/IMPALA-9489 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build needs > some attention before that can happen, per IMPALA-7825, and also conversation > I had with [~stakiar_impala_496e]. > However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python > files, and we should use those by default. It turns out that being stuck with > thrift-0.9.3 is a major impediment to achieving python 3 compatibility for > our python stack. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9489) Have impala-python env and impala-shell use available thrift-0.11.0 files
[ https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057492#comment-17057492 ] David Knupp commented on IMPALA-9489: - Review available at https://gerrit.cloudera.org/c/15417/ > Have impala-python env and impala-shell use available thrift-0.11.0 files > - > > Key: IMPALA-9489 > URL: https://issues.apache.org/jira/browse/IMPALA-9489 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build needs > some attention before that can happen, per IMPALA-7825, and also conversation > I had with [~stakiar_impala_496e]. > However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python > files, and we should use those by default. It turns out that being stuck with > thrift-0.9.3 is a major impediment to achieving python 3 compatibility for > our python stack. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9489) Have impala-python env and impala-shell use available thrift-0.11.0 files
[ https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp updated IMPALA-9489: Description: Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build needs some attention before that can happen, per IMPALA-7825, and also conversation I had with [~stakiar_impala_496e]. However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python files, and we should use those by default. It turns out that being stuck with thrift-0.9.3 is a major impediment to achieving python 3 compatibility for our python stack. was: Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build process needs some attention before that can happen, per IMPALA-7825, and a also conversation I had with [~stakiar_impala_496e]. However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python files, and we should use those by default. It turns out that being stuck with thrift-0.9.3 is a major impediment to achieving python 3 compatibility for our python stack. > Have impala-python env and impala-shell use available thrift-0.11.0 files > - > > Key: IMPALA-9489 > URL: https://issues.apache.org/jira/browse/IMPALA-9489 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build needs > some attention before that can happen, per IMPALA-7825, and also conversation > I had with [~stakiar_impala_496e]. > However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python > files, and we should use those by default. It turns out that being stuck with > thrift-0.9.3 is a major impediment to achieving python 3 compatibility for > our python stack. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9489) Have impala-python env and impala-shell use available thrift-0.11.0 files
[ https://issues.apache.org/jira/browse/IMPALA-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057488#comment-17057488 ] David Knupp commented on IMPALA-9489: - Cc: [~tarmstr...@cloudera.com], [~joemcdonnell], [~lenisha]. > Have impala-python env and impala-shell use available thrift-0.11.0 files > - > > Key: IMPALA-9489 > URL: https://issues.apache.org/jira/browse/IMPALA-9489 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build process > needs some attention before that can happen, per IMPALA-7825, and a also > conversation I had with [~stakiar_impala_496e]. > However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python > files, and we should use those by default. It turns out that being stuck with > thrift-0.9.3 is a major impediment to achieving python 3 compatibility for > our python stack. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9489) Have impala-python env and impala-shell use available thrift-0.11.0 files
David Knupp created IMPALA-9489: --- Summary: Have impala-python env and impala-shell use available thrift-0.11.0 files Key: IMPALA-9489 URL: https://issues.apache.org/jira/browse/IMPALA-9489 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.4.0 Reporter: David Knupp Assignee: David Knupp Apparently, we can't simply kick thrift-0.9.3 to the curb. Our build process needs some attention before that can happen, per IMPALA-7825, and a also conversation I had with [~stakiar_impala_496e]. However, with IMPALA-7924 resolved, we do have access to thrift-0.11.0 python files, and we should use those by default. It turns out that being stuck with thrift-0.9.3 is a major impediment to achieving python 3 compatibility for our python stack. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9436) impala-shell is very slow for large query text sizes
[ https://issues.apache.org/jira/browse/IMPALA-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp reassigned IMPALA-9436: --- Assignee: David Knupp > impala-shell is very slow for large query text sizes > > > Key: IMPALA-9436 > URL: https://issues.apache.org/jira/browse/IMPALA-9436 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: Thomas Tauber-Marshall >Assignee: David Knupp >Priority: Critical > > In working on better support for large sql queries in IMPALA-9414, I found > that impala-shell is very slow at processing large query sizes. > To test this, I generated a sql file of 1MB that refers to a non-existent > table (so that the time to run the query would be negligible). Running this > query file with impala-shell on my local machine takes about 20s, of which > about 13s are spent in parse_query_text(), which uses some sqlparse functions > to try to split the query text into multiple queries. > This seems like an unreasonable overhead and could definitely be improved. > Some ideas for how to do that: > 1. Be more clever with our use of sqlparse to get better perf. This probably > has limited value (eg. strip_comments() already tries to be very clever but > is still pretty slow) > 2. Find a different python library for sql parsing that is faster (this may > not exist). > 3. Add some C++ into the shell instead of always doing everything in pure > python (not sure how easy/convenient this is to integrate with the shell > packaging) > 4. Try to write our own sql parsing code, which could be optimized for the > small number of things we need actually need, eg. we don't need full > tokenization just splitting of multiple queries (likely to be bug-prone) > 5. Do some simple hacks, such as skipping the query splitting entirely if > there isn't a ';' in the query text (this would leave some unfortunate perf > cliffs, eg. add a ';' to a string literal in your query and suddenly > everything gets a lot slower) > 6. Add an interface in Impala that allows submitting of multiple queries at > once, eg ExecuteStatements(), which returns a list of query_ids. (might be a > lot of work to modify impala-server, the parser, etc. to support this) > 7. Add an interface in Impala that allows submitting of query text, then > parses it and returns it in split form without actually executing it, which > would limit the amount of changes needed vs. option 6 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9436) impala-shell is very slow for large query text sizes
[ https://issues.apache.org/jira/browse/IMPALA-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp reassigned IMPALA-9436: --- Assignee: (was: David Knupp) > impala-shell is very slow for large query text sizes > > > Key: IMPALA-9436 > URL: https://issues.apache.org/jira/browse/IMPALA-9436 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: Thomas Tauber-Marshall >Priority: Critical > > In working on better support for large sql queries in IMPALA-9414, I found > that impala-shell is very slow at processing large query sizes. > To test this, I generated a sql file of 1MB that refers to a non-existent > table (so that the time to run the query would be negligible). Running this > query file with impala-shell on my local machine takes about 20s, of which > about 13s are spent in parse_query_text(), which uses some sqlparse functions > to try to split the query text into multiple queries. > This seems like an unreasonable overhead and could definitely be improved. > Some ideas for how to do that: > 1. Be more clever with our use of sqlparse to get better perf. This probably > has limited value (eg. strip_comments() already tries to be very clever but > is still pretty slow) > 2. Find a different python library for sql parsing that is faster (this may > not exist). > 3. Add some C++ into the shell instead of always doing everything in pure > python (not sure how easy/convenient this is to integrate with the shell > packaging) > 4. Try to write our own sql parsing code, which could be optimized for the > small number of things we need actually need, eg. we don't need full > tokenization just splitting of multiple queries (likely to be bug-prone) > 5. Do some simple hacks, such as skipping the query splitting entirely if > there isn't a ';' in the query text (this would leave some unfortunate perf > cliffs, eg. add a ';' to a string literal in your query and suddenly > everything gets a lot slower) > 6. Add an interface in Impala that allows submitting of multiple queries at > once, eg ExecuteStatements(), which returns a list of query_ids. (might be a > lot of work to modify impala-server, the parser, etc. to support this) > 7. Add an interface in Impala that allows submitting of query text, then > parses it and returns it in split form without actually executing it, which > would limit the amount of changes needed vs. option 6 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9436) impala-shell is very slow for large query text sizes
[ https://issues.apache.org/jira/browse/IMPALA-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17047835#comment-17047835 ] David Knupp commented on IMPALA-9436: - For what it's worth, I did look into updating sqlparse as part of the fix for IMPALA-3343, but there were some issues re: backwards compatibility between python2 and python3, IIRC. It wasn't as straightforward as one would have hoped. I had planned to get the patch for IMPALA-3343 in first, and then figure out the sqlparse problem. > impala-shell is very slow for large query text sizes > > > Key: IMPALA-9436 > URL: https://issues.apache.org/jira/browse/IMPALA-9436 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Affects Versions: Impala 3.4.0 >Reporter: Thomas Tauber-Marshall >Priority: Critical > > In working on better support for large sql queries in IMPALA-9414, I found > that impala-shell is very slow at processing large query sizes. > To test this, I generated a sql file of 1MB that refers to a non-existent > table (so that the time to run the query would be negligible). Running this > query file with impala-shell on my local machine takes about 20s, of which > about 13s are spent in parse_query_text(), which uses some sqlparse functions > to try to split the query text into multiple queries. > This seems like an unreasonable overhead and could definitely be improved. > Some ideas for how to do that: > 1. Be more clever with our use of sqlparse to get better perf. This probably > has limited value (eg. strip_comments() already tries to be very clever but > is still pretty slow) > 2. Find a different python library for sql parsing that is faster (this may > not exist). > 3. Add some C++ into the shell instead of always doing everything in pure > python (not sure how easy/convenient this is to integrate with the shell > packaging) > 4. Try to write our own sql parsing code, which could be optimized for the > small number of things we need actually need, eg. we don't need full > tokenization just splitting of multiple queries (likely to be bug-prone) > 5. Do some simple hacks, such as skipping the query splitting entirely if > there isn't a ';' in the query text (this would leave some unfortunate perf > cliffs, eg. add a ';' to a string literal in your query and suddenly > everything gets a lot slower) > 6. Add an interface in Impala that allows submitting of multiple queries at > once, eg ExecuteStatements(), which returns a list of query_ids. (might be a > lot of work to modify impala-server, the parser, etc. to support this) > 7. Add an interface in Impala that allows submitting of query text, then > parses it and returns it in split form without actually executing it, which > would limit the amount of changes needed vs. option 6 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9424) Add six python library to shell/ext-py
[ https://issues.apache.org/jira/browse/IMPALA-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9424. - Resolution: Fixed > Add six python library to shell/ext-py > -- > > Key: IMPALA-9424 > URL: https://issues.apache.org/jira/browse/IMPALA-9424 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: David Knupp >Priority: Major > > A couple of impala-shell changes that are coming in the near future > (thrift_sasl update, possible changes to THttpClient, python 3 support) will > require the six python library. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9424) Add six python library to shell/ext-py
David Knupp created IMPALA-9424: --- Summary: Add six python library to shell/ext-py Key: IMPALA-9424 URL: https://issues.apache.org/jira/browse/IMPALA-9424 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.4.0 Reporter: David Knupp A couple of impala-shell changes that are coming in the near future (thrift_sasl update, possible changes to THttpClient, python 3 support) will require the six python library. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-3343) Impala-shell compatibility with python 3
[ https://issues.apache.org/jira/browse/IMPALA-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044051#comment-17044051 ] David Knupp commented on IMPALA-3343: - [~joemcdonnell] -- assuming no major issues are found during code review and testing, I'm hoping to get it wrapped up in the next... week or two...? > Impala-shell compatibility with python 3 > > > Key: IMPALA-3343 > URL: https://issues.apache.org/jira/browse/IMPALA-3343 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 2.5.0 >Reporter: Peter Ebert >Assignee: David Knupp >Priority: Critical > > Installed Anaconda package and python 3, Impala shell has errors and will not > run in the python 3 environment. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8373) impala-shell should support and be tested with Python 3
[ https://issues.apache.org/jira/browse/IMPALA-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032653#comment-17032653 ] David Knupp commented on IMPALA-8373: - Patch is ready for full review (no longer considered WIP). > impala-shell should support and be tested with Python 3 > --- > > Key: IMPALA-8373 > URL: https://issues.apache.org/jira/browse/IMPALA-8373 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Reporter: Tim Armstrong >Assignee: David Knupp >Priority: Major > Labels: shell > > Users are running into this issue - > https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/I-have-installed-the-Impala-using-the-CM-but-I-m-unable-to/m-p/88482#M5490%3Feid=1=1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-3343) Impala-shell compatibility with python 3
[ https://issues.apache.org/jira/browse/IMPALA-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032652#comment-17032652 ] David Knupp commented on IMPALA-3343: - Patch is ready for full review (no longer considered WIP). > Impala-shell compatibility with python 3 > > > Key: IMPALA-3343 > URL: https://issues.apache.org/jira/browse/IMPALA-3343 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 2.5.0 >Reporter: Peter Ebert >Assignee: David Knupp >Priority: Critical > > Installed Anaconda package and python 3, Impala shell has errors and will not > run in the python 3 environment. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8508) Use Python 3 from toolchain for impala-python
[ https://issues.apache.org/jira/browse/IMPALA-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027044#comment-17027044 ] David Knupp commented on IMPALA-8508: - Aside from the impala-shell, is there any other end user facing python? I don't think so, right? AFAIK, the remaining python dependency is the test framework. > Use Python 3 from toolchain for impala-python > - > > Key: IMPALA-8508 > URL: https://issues.apache.org/jira/browse/IMPALA-8508 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Tim Armstrong >Priority: Major > Attachments: > 0001-WIP-IMPALA-8508-download-Python-2.7-from-toolchain-i.patch > > > We should standardise on a single python version to use for tests and other > infrastructure. Python 2.7 is going EOL soon. > I started adding it to the toolchain - https://gerrit.cloudera.org/#/c/14161/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-3343) Impala-shell compatibility with python 3
[ https://issues.apache.org/jira/browse/IMPALA-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026279#comment-17026279 ] David Knupp commented on IMPALA-3343: - WIP patch is up for review: https://gerrit.cloudera.org/c/15132/ > Impala-shell compatibility with python 3 > > > Key: IMPALA-3343 > URL: https://issues.apache.org/jira/browse/IMPALA-3343 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 2.5.0 >Reporter: Peter Ebert >Assignee: David Knupp >Priority: Critical > > Installed Anaconda package and python 3, Impala shell has errors and will not > run in the python 3 environment. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8373) impala-shell should support and be tested with Python 3
[ https://issues.apache.org/jira/browse/IMPALA-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp updated IMPALA-8373: WIP patch is up for review: https://gerrit.cloudera.org/c/15132/ > impala-shell should support and be tested with Python 3 > --- > > Key: IMPALA-8373 > URL: https://issues.apache.org/jira/browse/IMPALA-8373 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Reporter: Tim Armstrong >Assignee: David Knupp >Priority: Major > Labels: shell > > Users are running into this issue - > https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/I-have-installed-the-Impala-using-the-CM-but-I-m-unable-to/m-p/88482#M5490%3Feid=1=1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8373) impala-shell should support and be tested with Python 3
[ https://issues.apache.org/jira/browse/IMPALA-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8373 started by David Knupp. --- > impala-shell should support and be tested with Python 3 > --- > > Key: IMPALA-8373 > URL: https://issues.apache.org/jira/browse/IMPALA-8373 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Reporter: Tim Armstrong >Assignee: David Knupp >Priority: Major > Labels: shell > > Users are running into this issue - > https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/I-have-installed-the-Impala-using-the-CM-but-I-m-unable-to/m-p/88482#M5490%3Feid=1=1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-3343) Impala-shell compatibility with python 3
[ https://issues.apache.org/jira/browse/IMPALA-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-3343 started by David Knupp. --- > Impala-shell compatibility with python 3 > > > Key: IMPALA-3343 > URL: https://issues.apache.org/jira/browse/IMPALA-3343 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 2.5.0 >Reporter: Peter Ebert >Assignee: David Knupp >Priority: Critical > > Installed Anaconda package and python 3, Impala shell has errors and will not > run in the python 3 environment. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9157) TestAuthorizationProvider.test_invalid_provider_flag fails due to Python 2.6 incompatible code
[
https://issues.apache.org/jira/browse/IMPALA-9157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980419#comment-16980419
]
David Knupp commented on IMPALA-9157:
-
Patch: https://gerrit.cloudera.org/c/14788/
> TestAuthorizationProvider.test_invalid_provider_flag fails due to Python 2.6
> incompatible code
> --
>
> Key: IMPALA-9157
> URL: https://issues.apache.org/jira/browse/IMPALA-9157
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.4.0
>Reporter: Joe McDonnell
>Assignee: David Knupp
>Priority: Blocker
> Labels: broken-build
>
> Our Centos 6 builds use Python 2.6, which means that it doesn't have
> check_output (added in Python 2.7). This causes test failures in
> test_provider.py:
>
> {noformat}
> authorization/test_provider.py:70: in setup_method
> self.pre_test_cores = set([f for f in possible_cores if is_core_dump(f)])
> ../lib/python/impala_py_lib/helpers.py:64: in is_core_dump
> file_std_out = exec_local_command("file %s" % file_path)
> ../lib/python/impala_py_lib/helpers.py:34: in exec_local_command
> return subprocess.check_output(cmd.split())
> E AttributeError: 'module' object has no attribute 'check_output'{noformat}
> This comes from the new code to handle intentional core dumps:
>
> [https://github.com/apache/impala/blob/master/lib/python/impala_py_lib/helpers.py#L34]
> {noformat}
> def exec_local_command(cmd):
> """ Executes a command for the local bash shell and return stdout as a
> string.
> Args:
> cmd: command as a string
> Return:
> STDOUT
> """
> return subprocess.check_output(cmd.split()){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9157) TestAuthorizationProvider.test_invalid_provider_flag fails due to Python 2.6 incompatible code
[
https://issues.apache.org/jira/browse/IMPALA-9157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9157 started by David Knupp.
---
> TestAuthorizationProvider.test_invalid_provider_flag fails due to Python 2.6
> incompatible code
> --
>
> Key: IMPALA-9157
> URL: https://issues.apache.org/jira/browse/IMPALA-9157
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.4.0
>Reporter: Joe McDonnell
>Assignee: David Knupp
>Priority: Blocker
> Labels: broken-build
>
> Our Centos 6 builds use Python 2.6, which means that it doesn't have
> check_output (added in Python 2.7). This causes test failures in
> test_provider.py:
>
> {noformat}
> authorization/test_provider.py:70: in setup_method
> self.pre_test_cores = set([f for f in possible_cores if is_core_dump(f)])
> ../lib/python/impala_py_lib/helpers.py:64: in is_core_dump
> file_std_out = exec_local_command("file %s" % file_path)
> ../lib/python/impala_py_lib/helpers.py:34: in exec_local_command
> return subprocess.check_output(cmd.split())
> E AttributeError: 'module' object has no attribute 'check_output'{noformat}
> This comes from the new code to handle intentional core dumps:
>
> [https://github.com/apache/impala/blob/master/lib/python/impala_py_lib/helpers.py#L34]
> {noformat}
> def exec_local_command(cmd):
> """ Executes a command for the local bash shell and return stdout as a
> string.
> Args:
> cmd: command as a string
> Return:
> STDOUT
> """
> return subprocess.check_output(cmd.split()){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9129) Provide a way for negative tests to remove intentionally generated core dumps
[ https://issues.apache.org/jira/browse/IMPALA-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-9129. - Resolution: Fixed > Provide a way for negative tests to remove intentionally generated core dumps > - > > Key: IMPALA-9129 > URL: https://issues.apache.org/jira/browse/IMPALA-9129 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > Occasionally, tests (esp. custom cluster tests) will inject an error or set > some invalid config, expecting Impala to generate a core dump. > We should have a general way for such files to delete the bogus core dumps, > otherwise they can complicate/confuse later triaging efforts of legitimate > test failures. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9129) Provide a way for negative tests to remove intentionally generated core dumps
[ https://issues.apache.org/jira/browse/IMPALA-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9129 started by David Knupp. --- > Provide a way for negative tests to remove intentionally generated core dumps > - > > Key: IMPALA-9129 > URL: https://issues.apache.org/jira/browse/IMPALA-9129 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > Occasionally, tests (esp. custom cluster tests) will inject an error or set > some invalid config, expecting Impala to generate a core dump. > We should have a general way for such files to delete the bogus core dumps, > otherwise they can complicate/confuse later triaging efforts of legitimate > test failures. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9129) Provide a way for negative tests to remove intentionally generated core dumps
[ https://issues.apache.org/jira/browse/IMPALA-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp reassigned IMPALA-9129: --- Assignee: David Knupp > Provide a way for negative tests to remove intentionally generated core dumps > - > > Key: IMPALA-9129 > URL: https://issues.apache.org/jira/browse/IMPALA-9129 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: David Knupp >Assignee: David Knupp >Priority: Major > > Occasionally, tests (esp. custom cluster tests) will perform some action, > expecting Impala to generate a core dump. > We should have a general way for such files to delete the bogus core dumps, > otherwise they can complicate/confuse later test triaging efforts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9129) Provide a way for negative tests to remove intentionally generated core dumps
David Knupp created IMPALA-9129: --- Summary: Provide a way for negative tests to remove intentionally generated core dumps Key: IMPALA-9129 URL: https://issues.apache.org/jira/browse/IMPALA-9129 Project: IMPALA Issue Type: Improvement Components: Infrastructure Reporter: David Knupp Occasionally, tests (esp. custom cluster tests) will perform some action, expecting Impala to generate a core dump. We should have a general way for such files to delete the bogus core dumps, otherwise they can complicate/confuse later test triaging efforts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-1071) Install impala-shell from PyPI
[ https://issues.apache.org/jira/browse/IMPALA-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Knupp resolved IMPALA-1071. - Resolution: Fixed > Install impala-shell from PyPI > -- > > Key: IMPALA-1071 > URL: https://issues.apache.org/jira/browse/IMPALA-1071 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Affects Versions: Impala 1.3.1 >Reporter: Shinya Okano >Assignee: David Knupp >Priority: Minor > Labels: shell > > I want to install impala-shell from PyPI (Python Package Index). > impala-shell appears to have been made with the Python module, a shell script > a little. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6808) Impala python code should be installed into infra/python/env like other packages
[
https://issues.apache.org/jira/browse/IMPALA-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923664#comment-16923664
]
David Knupp commented on IMPALA-6808:
-
As further evidence of what I was getting at, if you run Impala's local python,
by inspecting the various paths of top-level imported modules, it's easy way to
predicting where the code might be located, or how it became a part of the
Impala python environment.
{noformat}
>>> import sasl
>>> import requests
>>> import Logging
>>> import thrift
>>> requests
>>> sasl
>>> thrift
>>> Logging
{noformat}
> Impala python code should be installed into infra/python/env like other
> packages
>
>
> Key: IMPALA-6808
> URL: https://issues.apache.org/jira/browse/IMPALA-6808
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients, Infrastructure
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: David Knupp
>Assignee: David Knupp
>Priority: Major
>
> Impala/infra/python/env is the environment where necessary upstream python
> libraries and packages get installed -- e.g., the packages listed in
> https://github.com/apache/impala/blob/master/infra/python/deps/requirements.txt
> and other similar files.
> Impala's own internal python code (like the impala-shell, or the common test
> libraries that we rely upon) should be made available the same way -- as
> actual packages installed into the environment -- rather than by resorting to
> PYTHONPATH/sys.path sleight-of-hand, performed by such as
> bin/set-pythonpath.sh and bin/impala-python-common.sh.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8877) CatalogException during stress test: Table modified while operation was in progress
[
https://issues.apache.org/jira/browse/IMPALA-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-8877:
Summary: CatalogException during stress test: Table modified while
operation was in progress (was: CatalogException: Table modified while
operation was in progress, aborting execution.)
> CatalogException during stress test: Table modified while operation was
> in progress
> -
>
> Key: IMPALA-8877
> URL: https://issues.apache.org/jira/browse/IMPALA-8877
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.3.0
>Reporter: David Knupp
>Assignee: Vihang Karajgaonkar
>Priority: Critical
> Attachments: catalogd.INFO.tar.gz, impalad.INFO.tar.gz
>
>
> This was hit while running the stress tests to get a baseline on a deployed
> cluster.
> /* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> {noformat}
> Query (id=924a50178a5a6146:29d58a73)
> Summary
> Session ID: 5543fb9029e2b71f:f446381b1f59ed81
> Session Type: HIVESERVER2
> HiveServer2 Protocol Version: V6
> Start Time: 2019-08-19 01:26:07.292866000
> End Time: 2019-08-19 01:26:27.248053000
> Query Type: DDL
> Query State: EXCEPTION
> Query Status: CatalogException: Table
> 'tpcds_300_decimal_parquet.catalog_sales' was modified while operation was in
> progress, aborting execution.
> Impala Version: impalad version 3.3.0-SNAPSHOT RELEASE (build
> df3e7c051e2641524fc53a0cd07c2a14decd55f7)
> User: syst...@vpc.cloudera.com
> Connected User: syst...@vpc.cloudera.com
> Delegated User:
> Network Address: :::10.65.6.19:39174
> Default Db: tpcds_300_decimal_parquet
> Sql Statement: /* Mem: 12850 MB. Coordinator:
> quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> Coordinator: quasar-mzmnbe-6.vpc.cloudera.com:22000
> Query Options (set by configuration):
> ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> Query Options (set by configuration and planner):
> ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> DDL Type: COMPUTE_STATS
> Query Compilation
> Metadata of all 1 tables cached: 5.62s (5622372318)
> Analysis finished: 5.62s (5622560027)
> Authorization finished (noop): 5.62s (5622568284)
> Retried query planning due to inconsistent metadata 7 of 40 times:
> Catalog object TCatalogObject(type:TABLE, catalog_version:94204,
> table:TTable(db_name:tpcds_300_decimal_parquet, tbl_name:catalog_sales))
> changed version between accesses.: 5.95s (5949859598)
> Planning finished: 5.95s (5949861145)
> Query Timeline
> Query submitted: 0ns (0)
> Planning finished: 5.95s (5950024020)
> Child queries finished: 17.85s (17849072057)
> Rows available: 19.82s (19825080035)
> Unregister query: 19.95s (19955080560)
> Frontend
> - CatalogFetch.ColumnStats.Misses: 34 (34)
> - CatalogFetch.ColumnStats.Requests: 34 (34)
> - CatalogFetch.ColumnStats.Time: 0 (0)
> - CatalogFetch.Config.Hits: 1 (1)
> - CatalogFetch.Config.Requests: 1 (1)
> - CatalogFetch.Config.Time: 0 (0)
> - CatalogFetch.DatabaseList.Hits: 8 (8)
> - CatalogFetch.DatabaseList.Requests: 8 (8)
> - CatalogFetch.DatabaseList.Time: 0 (0)
> - CatalogFetch.PartitionLists.Misses: 1 (1)
> - CatalogFetch.PartitionLists.Requests: 1 (1)
> - CatalogFetch.PartitionLists.Time: 7 (7)
> - CatalogFetch.Partitions.Hits: 1837 (1837)
> - CatalogFetch.Partitions.Misses: 1837 (1837)
> - CatalogFetch.Partitions.Requests: 3674 (3674)
> - CatalogFetch.Partitions.Time: 325 (325)
> - CatalogFetch.RPCs.Bytes: 4.7 MiB (4936030)
> - CatalogFetch.RPCs.Requests: 22 (22)
> - CatalogFetch.RPCs.Time: 343 (343)
> - CatalogFetch.TableNames.Hits: 4 (4)
> - CatalogFetch.TableNames.Misses: 4 (4)
> - CatalogFetch.TableNames.Requests: 8 (8)
> - CatalogFetch.TableNames.Time: 0 (0)
> - CatalogFetch.Tables.Misses: 8 (8)
> - CatalogFetch.Tables.Requests: 8 (8)
> - CatalogFetch.Tables.Time: 74 (74)
> - InactiveTotalTime: 0ns (0)
> - TotalTime: 0ns (0)
> ImpalaServer
> - CatalogOpExecTimer: 1.97s (1972007962)
> - ClientFetchWaitTimer: 0ns (0)
> - InactiveTotalTime: 0ns (0)
> - RowMaterializationTimer: 0ns (0)
> - TotalTime: 0ns (0)
> Child Queries
> Table Stats Query
[jira] [Commented] (IMPALA-8873) LdapImpalaShellTest.testShellLdapAuth failed because Python version is too old
[
https://issues.apache.org/jira/browse/IMPALA-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910656#comment-16910656
]
David Knupp commented on IMPALA-8873:
-
[~bharathv] -- FYI, I think this fails with the ssl module in the python stdlib
for any version < 2.7.9.
> LdapImpalaShellTest.testShellLdapAuth failed because Python version is too old
> --
>
> Key: IMPALA-8873
> URL: https://issues.apache.org/jira/browse/IMPALA-8873
> Project: IMPALA
> Issue Type: Bug
> Environment: CentOS 6
>Reporter: Fang-Yu Rao
>Assignee: bharath v
>Priority: Blocker
> Labels: broken-build
>
> The recent build of impala-cdh6.x-exhaustive-centos6-shard1 failed due to a
> failed test `
> org.apache.impala.customcluster.LdapImpalaShellTest.testShellLdapAuth`,
> which is recently updated due to IMPALA-8828.
> According to the error messages provided in the following, it seems that the
> Python version is too old.
> {code:java}
> Starting Impala Shell using LDAP-based authentication
> Python version too old. SSLContext not supported.
> Error connecting: NotImplementedError,
> Not connected to Impala, could not execute queries.
> expected:<0> but was:<1>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-8877) CatalogException: Table modified while operation was in progress, aborting execution.
[
https://issues.apache.org/jira/browse/IMPALA-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910632#comment-16910632
]
David Knupp edited comment on IMPALA-8877 at 8/19/19 6:24 PM:
--
[~vihangk1] -- assigning to you as a best guess. Please reassign as needed.
was (Author: dknupp):
[~vihangk1] -- assigning to you as my best guess.
> CatalogException: Table modified while operation was in progress,
> aborting execution.
> ---
>
> Key: IMPALA-8877
> URL: https://issues.apache.org/jira/browse/IMPALA-8877
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.3.0
>Reporter: David Knupp
>Assignee: Vihang Karajgaonkar
>Priority: Critical
> Attachments: catalogd.INFO.tar.gz, impalad.INFO.tar.gz
>
>
> This was hit while running the stress tests to get a baseline on a deployed
> cluster.
> /* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> {noformat}
> Query (id=924a50178a5a6146:29d58a73)
> Summary
> Session ID: 5543fb9029e2b71f:f446381b1f59ed81
> Session Type: HIVESERVER2
> HiveServer2 Protocol Version: V6
> Start Time: 2019-08-19 01:26:07.292866000
> End Time: 2019-08-19 01:26:27.248053000
> Query Type: DDL
> Query State: EXCEPTION
> Query Status: CatalogException: Table
> 'tpcds_300_decimal_parquet.catalog_sales' was modified while operation was in
> progress, aborting execution.
> Impala Version: impalad version 3.3.0-SNAPSHOT RELEASE (build
> df3e7c051e2641524fc53a0cd07c2a14decd55f7)
> User: syst...@vpc.cloudera.com
> Connected User: syst...@vpc.cloudera.com
> Delegated User:
> Network Address: :::10.65.6.19:39174
> Default Db: tpcds_300_decimal_parquet
> Sql Statement: /* Mem: 12850 MB. Coordinator:
> quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> Coordinator: quasar-mzmnbe-6.vpc.cloudera.com:22000
> Query Options (set by configuration):
> ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> Query Options (set by configuration and planner):
> ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> DDL Type: COMPUTE_STATS
> Query Compilation
> Metadata of all 1 tables cached: 5.62s (5622372318)
> Analysis finished: 5.62s (5622560027)
> Authorization finished (noop): 5.62s (5622568284)
> Retried query planning due to inconsistent metadata 7 of 40 times:
> Catalog object TCatalogObject(type:TABLE, catalog_version:94204,
> table:TTable(db_name:tpcds_300_decimal_parquet, tbl_name:catalog_sales))
> changed version between accesses.: 5.95s (5949859598)
> Planning finished: 5.95s (5949861145)
> Query Timeline
> Query submitted: 0ns (0)
> Planning finished: 5.95s (5950024020)
> Child queries finished: 17.85s (17849072057)
> Rows available: 19.82s (19825080035)
> Unregister query: 19.95s (19955080560)
> Frontend
> - CatalogFetch.ColumnStats.Misses: 34 (34)
> - CatalogFetch.ColumnStats.Requests: 34 (34)
> - CatalogFetch.ColumnStats.Time: 0 (0)
> - CatalogFetch.Config.Hits: 1 (1)
> - CatalogFetch.Config.Requests: 1 (1)
> - CatalogFetch.Config.Time: 0 (0)
> - CatalogFetch.DatabaseList.Hits: 8 (8)
> - CatalogFetch.DatabaseList.Requests: 8 (8)
> - CatalogFetch.DatabaseList.Time: 0 (0)
> - CatalogFetch.PartitionLists.Misses: 1 (1)
> - CatalogFetch.PartitionLists.Requests: 1 (1)
> - CatalogFetch.PartitionLists.Time: 7 (7)
> - CatalogFetch.Partitions.Hits: 1837 (1837)
> - CatalogFetch.Partitions.Misses: 1837 (1837)
> - CatalogFetch.Partitions.Requests: 3674 (3674)
> - CatalogFetch.Partitions.Time: 325 (325)
> - CatalogFetch.RPCs.Bytes: 4.7 MiB (4936030)
> - CatalogFetch.RPCs.Requests: 22 (22)
> - CatalogFetch.RPCs.Time: 343 (343)
> - CatalogFetch.TableNames.Hits: 4 (4)
> - CatalogFetch.TableNames.Misses: 4 (4)
> - CatalogFetch.TableNames.Requests: 8 (8)
> - CatalogFetch.TableNames.Time: 0 (0)
> - CatalogFetch.Tables.Misses: 8 (8)
> - CatalogFetch.Tables.Requests: 8 (8)
> - CatalogFetch.Tables.Time: 74 (74)
> - InactiveTotalTime: 0ns (0)
> - TotalTime: 0ns (0)
> ImpalaServer
> - CatalogOpExecTimer: 1.97s (1972007962)
> - ClientFetchWaitTimer: 0ns (0)
> - InactiveTotalTime: 0ns (0)
> - RowMaterializationTimer: 0ns (0)
> - TotalTime: 0ns (0)
> Child Queries
>
[jira] [Commented] (IMPALA-8877) CatalogException: Table modified while operation was in progress, aborting execution.
[
https://issues.apache.org/jira/browse/IMPALA-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910632#comment-16910632
]
David Knupp commented on IMPALA-8877:
-
[~vihangk1] -- assigning to you as my best guess.
> CatalogException: Table modified while operation was in progress,
> aborting execution.
> ---
>
> Key: IMPALA-8877
> URL: https://issues.apache.org/jira/browse/IMPALA-8877
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.3.0
>Reporter: David Knupp
>Assignee: Vihang Karajgaonkar
>Priority: Critical
> Attachments: catalogd.INFO.tar.gz, impalad.INFO.tar.gz
>
>
> This was hit while running the stress tests to get a baseline on a deployed
> cluster.
> /* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> {noformat}
> Query (id=924a50178a5a6146:29d58a73)
> Summary
> Session ID: 5543fb9029e2b71f:f446381b1f59ed81
> Session Type: HIVESERVER2
> HiveServer2 Protocol Version: V6
> Start Time: 2019-08-19 01:26:07.292866000
> End Time: 2019-08-19 01:26:27.248053000
> Query Type: DDL
> Query State: EXCEPTION
> Query Status: CatalogException: Table
> 'tpcds_300_decimal_parquet.catalog_sales' was modified while operation was in
> progress, aborting execution.
> Impala Version: impalad version 3.3.0-SNAPSHOT RELEASE (build
> df3e7c051e2641524fc53a0cd07c2a14decd55f7)
> User: syst...@vpc.cloudera.com
> Connected User: syst...@vpc.cloudera.com
> Delegated User:
> Network Address: :::10.65.6.19:39174
> Default Db: tpcds_300_decimal_parquet
> Sql Statement: /* Mem: 12850 MB. Coordinator:
> quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> Coordinator: quasar-mzmnbe-6.vpc.cloudera.com:22000
> Query Options (set by configuration):
> ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> Query Options (set by configuration and planner):
> ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> DDL Type: COMPUTE_STATS
> Query Compilation
> Metadata of all 1 tables cached: 5.62s (5622372318)
> Analysis finished: 5.62s (5622560027)
> Authorization finished (noop): 5.62s (5622568284)
> Retried query planning due to inconsistent metadata 7 of 40 times:
> Catalog object TCatalogObject(type:TABLE, catalog_version:94204,
> table:TTable(db_name:tpcds_300_decimal_parquet, tbl_name:catalog_sales))
> changed version between accesses.: 5.95s (5949859598)
> Planning finished: 5.95s (5949861145)
> Query Timeline
> Query submitted: 0ns (0)
> Planning finished: 5.95s (5950024020)
> Child queries finished: 17.85s (17849072057)
> Rows available: 19.82s (19825080035)
> Unregister query: 19.95s (19955080560)
> Frontend
> - CatalogFetch.ColumnStats.Misses: 34 (34)
> - CatalogFetch.ColumnStats.Requests: 34 (34)
> - CatalogFetch.ColumnStats.Time: 0 (0)
> - CatalogFetch.Config.Hits: 1 (1)
> - CatalogFetch.Config.Requests: 1 (1)
> - CatalogFetch.Config.Time: 0 (0)
> - CatalogFetch.DatabaseList.Hits: 8 (8)
> - CatalogFetch.DatabaseList.Requests: 8 (8)
> - CatalogFetch.DatabaseList.Time: 0 (0)
> - CatalogFetch.PartitionLists.Misses: 1 (1)
> - CatalogFetch.PartitionLists.Requests: 1 (1)
> - CatalogFetch.PartitionLists.Time: 7 (7)
> - CatalogFetch.Partitions.Hits: 1837 (1837)
> - CatalogFetch.Partitions.Misses: 1837 (1837)
> - CatalogFetch.Partitions.Requests: 3674 (3674)
> - CatalogFetch.Partitions.Time: 325 (325)
> - CatalogFetch.RPCs.Bytes: 4.7 MiB (4936030)
> - CatalogFetch.RPCs.Requests: 22 (22)
> - CatalogFetch.RPCs.Time: 343 (343)
> - CatalogFetch.TableNames.Hits: 4 (4)
> - CatalogFetch.TableNames.Misses: 4 (4)
> - CatalogFetch.TableNames.Requests: 8 (8)
> - CatalogFetch.TableNames.Time: 0 (0)
> - CatalogFetch.Tables.Misses: 8 (8)
> - CatalogFetch.Tables.Requests: 8 (8)
> - CatalogFetch.Tables.Time: 74 (74)
> - InactiveTotalTime: 0ns (0)
> - TotalTime: 0ns (0)
> ImpalaServer
> - CatalogOpExecTimer: 1.97s (1972007962)
> - ClientFetchWaitTimer: 0ns (0)
> - InactiveTotalTime: 0ns (0)
> - RowMaterializationTimer: 0ns (0)
> - TotalTime: 0ns (0)
> Child Queries
> Table Stats Query (id=db4821e4aa5bb04d:d4a5ae45)
> Column Stats Query (id=0444367557e3496d:f9435111)
> {noformat}
--
This message
[jira] [Assigned] (IMPALA-8877) CatalogException: Table modified while operation was in progress, aborting execution.
[
https://issues.apache.org/jira/browse/IMPALA-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp reassigned IMPALA-8877:
---
Assignee: Vihang Karajgaonkar (was: David Knupp)
> CatalogException: Table modified while operation was in progress,
> aborting execution.
> ---
>
> Key: IMPALA-8877
> URL: https://issues.apache.org/jira/browse/IMPALA-8877
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.3.0
>Reporter: David Knupp
>Assignee: Vihang Karajgaonkar
>Priority: Critical
> Attachments: catalogd.INFO.tar.gz, impalad.INFO.tar.gz
>
>
> This was hit while running the stress tests to get a baseline on a deployed
> cluster.
> /* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> {noformat}
> Query (id=924a50178a5a6146:29d58a73)
> Summary
> Session ID: 5543fb9029e2b71f:f446381b1f59ed81
> Session Type: HIVESERVER2
> HiveServer2 Protocol Version: V6
> Start Time: 2019-08-19 01:26:07.292866000
> End Time: 2019-08-19 01:26:27.248053000
> Query Type: DDL
> Query State: EXCEPTION
> Query Status: CatalogException: Table
> 'tpcds_300_decimal_parquet.catalog_sales' was modified while operation was in
> progress, aborting execution.
> Impala Version: impalad version 3.3.0-SNAPSHOT RELEASE (build
> df3e7c051e2641524fc53a0cd07c2a14decd55f7)
> User: syst...@vpc.cloudera.com
> Connected User: syst...@vpc.cloudera.com
> Delegated User:
> Network Address: :::10.65.6.19:39174
> Default Db: tpcds_300_decimal_parquet
> Sql Statement: /* Mem: 12850 MB. Coordinator:
> quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> Coordinator: quasar-mzmnbe-6.vpc.cloudera.com:22000
> Query Options (set by configuration):
> ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> Query Options (set by configuration and planner):
> ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> DDL Type: COMPUTE_STATS
> Query Compilation
> Metadata of all 1 tables cached: 5.62s (5622372318)
> Analysis finished: 5.62s (5622560027)
> Authorization finished (noop): 5.62s (5622568284)
> Retried query planning due to inconsistent metadata 7 of 40 times:
> Catalog object TCatalogObject(type:TABLE, catalog_version:94204,
> table:TTable(db_name:tpcds_300_decimal_parquet, tbl_name:catalog_sales))
> changed version between accesses.: 5.95s (5949859598)
> Planning finished: 5.95s (5949861145)
> Query Timeline
> Query submitted: 0ns (0)
> Planning finished: 5.95s (5950024020)
> Child queries finished: 17.85s (17849072057)
> Rows available: 19.82s (19825080035)
> Unregister query: 19.95s (19955080560)
> Frontend
> - CatalogFetch.ColumnStats.Misses: 34 (34)
> - CatalogFetch.ColumnStats.Requests: 34 (34)
> - CatalogFetch.ColumnStats.Time: 0 (0)
> - CatalogFetch.Config.Hits: 1 (1)
> - CatalogFetch.Config.Requests: 1 (1)
> - CatalogFetch.Config.Time: 0 (0)
> - CatalogFetch.DatabaseList.Hits: 8 (8)
> - CatalogFetch.DatabaseList.Requests: 8 (8)
> - CatalogFetch.DatabaseList.Time: 0 (0)
> - CatalogFetch.PartitionLists.Misses: 1 (1)
> - CatalogFetch.PartitionLists.Requests: 1 (1)
> - CatalogFetch.PartitionLists.Time: 7 (7)
> - CatalogFetch.Partitions.Hits: 1837 (1837)
> - CatalogFetch.Partitions.Misses: 1837 (1837)
> - CatalogFetch.Partitions.Requests: 3674 (3674)
> - CatalogFetch.Partitions.Time: 325 (325)
> - CatalogFetch.RPCs.Bytes: 4.7 MiB (4936030)
> - CatalogFetch.RPCs.Requests: 22 (22)
> - CatalogFetch.RPCs.Time: 343 (343)
> - CatalogFetch.TableNames.Hits: 4 (4)
> - CatalogFetch.TableNames.Misses: 4 (4)
> - CatalogFetch.TableNames.Requests: 8 (8)
> - CatalogFetch.TableNames.Time: 0 (0)
> - CatalogFetch.Tables.Misses: 8 (8)
> - CatalogFetch.Tables.Requests: 8 (8)
> - CatalogFetch.Tables.Time: 74 (74)
> - InactiveTotalTime: 0ns (0)
> - TotalTime: 0ns (0)
> ImpalaServer
> - CatalogOpExecTimer: 1.97s (1972007962)
> - ClientFetchWaitTimer: 0ns (0)
> - InactiveTotalTime: 0ns (0)
> - RowMaterializationTimer: 0ns (0)
> - TotalTime: 0ns (0)
> Child Queries
> Table Stats Query (id=db4821e4aa5bb04d:d4a5ae45)
> Column Stats Query (id=0444367557e3496d:f9435111)
> {noformat}
--
This message was sent by Atlassian Jira
[jira] [Created] (IMPALA-8877) CatalogException: Table modified while operation was in progress, aborting execution.
David Knupp created IMPALA-8877:
---
Summary: CatalogException: Table modified while operation
was in progress, aborting execution.
Key: IMPALA-8877
URL: https://issues.apache.org/jira/browse/IMPALA-8877
Project: IMPALA
Issue Type: Bug
Components: Catalog
Affects Versions: Impala 3.3.0
Reporter: David Knupp
Attachments: catalogd.INFO.tar.gz, impalad.INFO.tar.gz
This was hit while running the stress tests to get a baseline on a deployed
cluster.
/* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */
COMPUTE STATS catalog_sales
{noformat}
Query (id=924a50178a5a6146:29d58a73)
Summary
Session ID: 5543fb9029e2b71f:f446381b1f59ed81
Session Type: HIVESERVER2
HiveServer2 Protocol Version: V6
Start Time: 2019-08-19 01:26:07.292866000
End Time: 2019-08-19 01:26:27.248053000
Query Type: DDL
Query State: EXCEPTION
Query Status: CatalogException: Table
'tpcds_300_decimal_parquet.catalog_sales' was modified while operation was in
progress, aborting execution.
Impala Version: impalad version 3.3.0-SNAPSHOT RELEASE (build
df3e7c051e2641524fc53a0cd07c2a14decd55f7)
User: syst...@vpc.cloudera.com
Connected User: syst...@vpc.cloudera.com
Delegated User:
Network Address: :::10.65.6.19:39174
Default Db: tpcds_300_decimal_parquet
Sql Statement: /* Mem: 12850 MB. Coordinator:
quasar-mzmnbe-6.vpc.cloudera.com. */
COMPUTE STATS catalog_sales
Coordinator: quasar-mzmnbe-6.vpc.cloudera.com:22000
Query Options (set by configuration):
ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
Query Options (set by configuration and planner):
ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
DDL Type: COMPUTE_STATS
Query Compilation
Metadata of all 1 tables cached: 5.62s (5622372318)
Analysis finished: 5.62s (5622560027)
Authorization finished (noop): 5.62s (5622568284)
Retried query planning due to inconsistent metadata 7 of 40 times:
Catalog object TCatalogObject(type:TABLE, catalog_version:94204,
table:TTable(db_name:tpcds_300_decimal_parquet, tbl_name:catalog_sales))
changed version between accesses.: 5.95s (5949859598)
Planning finished: 5.95s (5949861145)
Query Timeline
Query submitted: 0ns (0)
Planning finished: 5.95s (5950024020)
Child queries finished: 17.85s (17849072057)
Rows available: 19.82s (19825080035)
Unregister query: 19.95s (19955080560)
Frontend
- CatalogFetch.ColumnStats.Misses: 34 (34)
- CatalogFetch.ColumnStats.Requests: 34 (34)
- CatalogFetch.ColumnStats.Time: 0 (0)
- CatalogFetch.Config.Hits: 1 (1)
- CatalogFetch.Config.Requests: 1 (1)
- CatalogFetch.Config.Time: 0 (0)
- CatalogFetch.DatabaseList.Hits: 8 (8)
- CatalogFetch.DatabaseList.Requests: 8 (8)
- CatalogFetch.DatabaseList.Time: 0 (0)
- CatalogFetch.PartitionLists.Misses: 1 (1)
- CatalogFetch.PartitionLists.Requests: 1 (1)
- CatalogFetch.PartitionLists.Time: 7 (7)
- CatalogFetch.Partitions.Hits: 1837 (1837)
- CatalogFetch.Partitions.Misses: 1837 (1837)
- CatalogFetch.Partitions.Requests: 3674 (3674)
- CatalogFetch.Partitions.Time: 325 (325)
- CatalogFetch.RPCs.Bytes: 4.7 MiB (4936030)
- CatalogFetch.RPCs.Requests: 22 (22)
- CatalogFetch.RPCs.Time: 343 (343)
- CatalogFetch.TableNames.Hits: 4 (4)
- CatalogFetch.TableNames.Misses: 4 (4)
- CatalogFetch.TableNames.Requests: 8 (8)
- CatalogFetch.TableNames.Time: 0 (0)
- CatalogFetch.Tables.Misses: 8 (8)
- CatalogFetch.Tables.Requests: 8 (8)
- CatalogFetch.Tables.Time: 74 (74)
- InactiveTotalTime: 0ns (0)
- TotalTime: 0ns (0)
ImpalaServer
- CatalogOpExecTimer: 1.97s (1972007962)
- ClientFetchWaitTimer: 0ns (0)
- InactiveTotalTime: 0ns (0)
- RowMaterializationTimer: 0ns (0)
- TotalTime: 0ns (0)
Child Queries
Table Stats Query (id=db4821e4aa5bb04d:d4a5ae45)
Column Stats Query (id=0444367557e3496d:f9435111)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8877) CatalogException: Table modified while operation was in progress, aborting execution.
[
https://issues.apache.org/jira/browse/IMPALA-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp reassigned IMPALA-8877:
---
Assignee: David Knupp
> CatalogException: Table modified while operation was in progress,
> aborting execution.
> ---
>
> Key: IMPALA-8877
> URL: https://issues.apache.org/jira/browse/IMPALA-8877
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.3.0
>Reporter: David Knupp
>Assignee: David Knupp
>Priority: Critical
> Attachments: catalogd.INFO.tar.gz, impalad.INFO.tar.gz
>
>
> This was hit while running the stress tests to get a baseline on a deployed
> cluster.
> /* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> {noformat}
> Query (id=924a50178a5a6146:29d58a73)
> Summary
> Session ID: 5543fb9029e2b71f:f446381b1f59ed81
> Session Type: HIVESERVER2
> HiveServer2 Protocol Version: V6
> Start Time: 2019-08-19 01:26:07.292866000
> End Time: 2019-08-19 01:26:27.248053000
> Query Type: DDL
> Query State: EXCEPTION
> Query Status: CatalogException: Table
> 'tpcds_300_decimal_parquet.catalog_sales' was modified while operation was in
> progress, aborting execution.
> Impala Version: impalad version 3.3.0-SNAPSHOT RELEASE (build
> df3e7c051e2641524fc53a0cd07c2a14decd55f7)
> User: syst...@vpc.cloudera.com
> Connected User: syst...@vpc.cloudera.com
> Delegated User:
> Network Address: :::10.65.6.19:39174
> Default Db: tpcds_300_decimal_parquet
> Sql Statement: /* Mem: 12850 MB. Coordinator:
> quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> Coordinator: quasar-mzmnbe-6.vpc.cloudera.com:22000
> Query Options (set by configuration):
> ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> Query Options (set by configuration and planner):
> ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> DDL Type: COMPUTE_STATS
> Query Compilation
> Metadata of all 1 tables cached: 5.62s (5622372318)
> Analysis finished: 5.62s (5622560027)
> Authorization finished (noop): 5.62s (5622568284)
> Retried query planning due to inconsistent metadata 7 of 40 times:
> Catalog object TCatalogObject(type:TABLE, catalog_version:94204,
> table:TTable(db_name:tpcds_300_decimal_parquet, tbl_name:catalog_sales))
> changed version between accesses.: 5.95s (5949859598)
> Planning finished: 5.95s (5949861145)
> Query Timeline
> Query submitted: 0ns (0)
> Planning finished: 5.95s (5950024020)
> Child queries finished: 17.85s (17849072057)
> Rows available: 19.82s (19825080035)
> Unregister query: 19.95s (19955080560)
> Frontend
> - CatalogFetch.ColumnStats.Misses: 34 (34)
> - CatalogFetch.ColumnStats.Requests: 34 (34)
> - CatalogFetch.ColumnStats.Time: 0 (0)
> - CatalogFetch.Config.Hits: 1 (1)
> - CatalogFetch.Config.Requests: 1 (1)
> - CatalogFetch.Config.Time: 0 (0)
> - CatalogFetch.DatabaseList.Hits: 8 (8)
> - CatalogFetch.DatabaseList.Requests: 8 (8)
> - CatalogFetch.DatabaseList.Time: 0 (0)
> - CatalogFetch.PartitionLists.Misses: 1 (1)
> - CatalogFetch.PartitionLists.Requests: 1 (1)
> - CatalogFetch.PartitionLists.Time: 7 (7)
> - CatalogFetch.Partitions.Hits: 1837 (1837)
> - CatalogFetch.Partitions.Misses: 1837 (1837)
> - CatalogFetch.Partitions.Requests: 3674 (3674)
> - CatalogFetch.Partitions.Time: 325 (325)
> - CatalogFetch.RPCs.Bytes: 4.7 MiB (4936030)
> - CatalogFetch.RPCs.Requests: 22 (22)
> - CatalogFetch.RPCs.Time: 343 (343)
> - CatalogFetch.TableNames.Hits: 4 (4)
> - CatalogFetch.TableNames.Misses: 4 (4)
> - CatalogFetch.TableNames.Requests: 8 (8)
> - CatalogFetch.TableNames.Time: 0 (0)
> - CatalogFetch.Tables.Misses: 8 (8)
> - CatalogFetch.Tables.Requests: 8 (8)
> - CatalogFetch.Tables.Time: 74 (74)
> - InactiveTotalTime: 0ns (0)
> - TotalTime: 0ns (0)
> ImpalaServer
> - CatalogOpExecTimer: 1.97s (1972007962)
> - ClientFetchWaitTimer: 0ns (0)
> - InactiveTotalTime: 0ns (0)
> - RowMaterializationTimer: 0ns (0)
> - TotalTime: 0ns (0)
> Child Queries
> Table Stats Query (id=db4821e4aa5bb04d:d4a5ae45)
> Column Stats Query (id=0444367557e3496d:f9435111)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (IMPALA-8553) Several tests failing with connection errors on deployed clusters
[
https://issues.apache.org/jira/browse/IMPALA-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868985#comment-16868985
]
David Knupp commented on IMPALA-8553:
-
Thanks for the fix, Tim.
> Several tests failing with connection errors on deployed clusters
> -
>
> Key: IMPALA-8553
> URL: https://issues.apache.org/jira/browse/IMPALA-8553
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Critical
> Fix For: Impala 3.3.0
>
>
> The errors look fairly similar. I suspect this commit introduced a regression:
> https://github.com/apache/impala/commit/79c5f875
> *Stacktrace*
> {noformat}
> metadata/test_hms_integration.py:66: in test_sanity
> if IMPALA_TEST_CLUSTER_PROPERTIES.is_catalog_v2_cluster():
> common/environ.py:307: in is_catalog_v2_cluster
> flags = self._get_flags_from_web_ui(web_ui_url)
> common/environ.py:295: in _get_flags_from_web_ui
> response = requests.get(impala_url + "/varz?json")
> ../infra/python/env/lib/python2.7/site-packages/requests/api.py:69: in get
> return request('get', url, params=params, **kwargs)
> ../infra/python/env/lib/python2.7/site-packages/requests/api.py:50: in request
> response = session.request(method=method, url=url, **kwargs)
> ../infra/python/env/lib/python2.7/site-packages/requests/sessions.py:465: in
> request
> resp = self.send(prep, **send_kwargs)
> ../infra/python/env/lib/python2.7/site-packages/requests/sessions.py:573: in
> send
> r = adapter.send(request, **kwargs)
> ../infra/python/env/lib/python2.7/site-packages/requests/adapters.py:415: in
> send
> raise ConnectionError(err, request=request)
> E ConnectionError: ('Connection aborted.', error(111, 'Connection refused'))
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-3336) psycopg2 as used by qgen prefetches rows, can consume too much memory
[
https://issues.apache.org/jira/browse/IMPALA-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847874#comment-16847874
]
David Knupp commented on IMPALA-3336:
-
Thanks! Useful information in general.
> psycopg2 as used by qgen prefetches rows, can consume too much memory
> -
>
> Key: IMPALA-3336
> URL: https://issues.apache.org/jira/browse/IMPALA-3336
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.6.0
>Reporter: Michael Brown
>Priority: Minor
>
> When I was experimenting with the query generator on EC2, I started having
> problems with the runner process being killed via kernel oom_killer. In
> checking in with Taras, he let me know this was happening to him as well. I
> need to figure out why that is happening so that the query generator can run.
> {noformat}
> 16:07:51 16:07:52 Query execution thread
> run_query_internal_i80swzzi4opv3q0p:hiveserver2[307]:Fetching up to 2000
> result rows
> 16:07:52 16:07:52 Query execution thread
> run_query_internal_i80swzzi4opv3q0p:hiveserver2[307]:Fetching up to 2000
> result rows
> 16:07:52 16:07:52 Query execution thread
> run_query_internal_i80swzzi4opv3q0p:hiveserver2[307]:Fetching up to 2000
> result rows
> 16:08:28 /tmp/hudson351019564031822959.sh: line 11: 4588 Killed
> ./tests/comparison/leopard/job.py
> 16:08:28 Build step 'Execute shell' marked build as failure
> 16:08:31 Finished: FAILURE
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8552) impala-shell tests break on deployed clusters if IMPALA_LOCAL_BUILD_VERSION is None
[
https://issues.apache.org/jira/browse/IMPALA-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841834#comment-16841834
]
David Knupp commented on IMPALA-8552:
-
It's hard to say. Given our current setup, I guess my gut would be to have
environ.py be the source of truth, since conftest.py already imports it? I
suppose both environ.py and create-load-data.sh (and friends) will need to
consult the same env variable though?
In a perfect world, I think the test harness really shouldn't care about remote
vs. local. Ideally, we'd just provide the test framework with endpoints/port
numbers of the necessary services, and 127.0.0.1: would simply be among
the possible options.
> impala-shell tests break on deployed clusters if IMPALA_LOCAL_BUILD_VERSION
> is None
> ---
>
> Key: IMPALA-8552
> URL: https://issues.apache.org/jira/browse/IMPALA-8552
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Critical
>
> This is a regression introduced by the commit:
> https://github.com/apache/impala/commit/b55d905
> *Stacktrace*
> {noformat}
> shell/test_shell_commandline.py:33: in
> from util import (get_impalad_host_port, assert_var_substitution,
> run_impala_shell_cmd,
> shell/util.py:42: in
> IMPALA_HOME, "shell/build", "impala-shell-" + IMPALA_LOCAL_BUILD_VERSION,
> E TypeError: cannot concatenate 'str' and 'NoneType' objects
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8552) impala-shell tests break on deployed clusters if IMPALA_LOCAL_BUILD_VERSION is None
[
https://issues.apache.org/jira/browse/IMPALA-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841793#comment-16841793
]
David Knupp commented on IMPALA-8552:
-
I was confused by this too. I think one of the problems is that our build
scripts and our test code are too intertwined. There are too many points of
entry, and it's nearly impossible to track all of it -- especially since we
tend to rely too heavily on arbitrary global env variables. In this case, the
code you're referencing was introduced relatively recently in
https://github.com/apache/impala/commit/691f9d9.
In this case, is_remote_cluster() seems to rely on IMPALA_REMOTE_URL, which was
added as part of the same commit.
{noformat}
$ git grep IMPALA_REMOTE_URL
tests/common/environ.py:IMPALA_REMOTE_URL = os.environ.get("IMPALA_REMOTE_URL",
"")
{noformat}
Elsewhere, an env variable called REMOTE_LOAD is used to determine whether data
load scripts are loading script locally or not.
Finally, in conftest.py, there's also a flag for {{--testing_remote_cluster}}
(and a SkipIf definition based upon it). There's nothing that coordinates any
of these inputs, or guarantees that they need to agree. It's a little maddening.
> impala-shell tests break on deployed clusters if IMPALA_LOCAL_BUILD_VERSION
> is None
> ---
>
> Key: IMPALA-8552
> URL: https://issues.apache.org/jira/browse/IMPALA-8552
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Critical
>
> This is a regression introduced by the commit:
> https://github.com/apache/impala/commit/b55d905
> *Stacktrace*
> {noformat}
> shell/test_shell_commandline.py:33: in
> from util import (get_impalad_host_port, assert_var_substitution,
> run_impala_shell_cmd,
> shell/util.py:42: in
> IMPALA_HOME, "shell/build", "impala-shell-" + IMPALA_LOCAL_BUILD_VERSION,
> E TypeError: cannot concatenate 'str' and 'NoneType' objects
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-8552) impala-shell tests break on deployed clusters if IMPALA_LOCAL_BUILD_VERSION is None
[
https://issues.apache.org/jira/browse/IMPALA-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841793#comment-16841793
]
David Knupp edited comment on IMPALA-8552 at 5/16/19 11:01 PM:
---
[~tarmstrong] – I was confused by this too. I think one of the problems is that
our build scripts and our test code are too intertwined. There are too many
points of entry, and it's nearly impossible to track all of it – especially
since we tend to rely too heavily on arbitrary global env variables. In this
case, the code you're referencing was introduced relatively recently in
[https://github.com/apache/impala/commit/691f9d9].
is_remote_cluster() seems to rely on IMPALA_REMOTE_URL, which was added as part
of the same commit.
{noformat}
$ git grep IMPALA_REMOTE_URL
tests/common/environ.py:IMPALA_REMOTE_URL = os.environ.get("IMPALA_REMOTE_URL",
"")
{noformat}
Elsewhere, an env variable called REMOTE_LOAD is used to determine whether data
load scripts are loading script locally or not.
Finally, in conftest.py, there's also a flag for {{--testing_remote_cluster}}
(and a SkipIf definition based upon it). There's nothing that coordinates any
of these inputs, or guarantees that they need to agree. It's a little maddening.
was (Author: dknupp):
[~tarmstrong] – I was confused by this too. I think one of the problems is that
our build scripts and our test code are too intertwined. There are too many
points of entry, and it's nearly impossible to track all of it – especially
since we tend to rely too heavily on arbitrary global env variables. In this
case, the code you're referencing was introduced relatively recently in
[https://github.com/apache/impala/commit/691f9d9].
In this case, is_remote_cluster() seems to rely on IMPALA_REMOTE_URL, which was
added as part of the same commit.
{noformat}
$ git grep IMPALA_REMOTE_URL
tests/common/environ.py:IMPALA_REMOTE_URL = os.environ.get("IMPALA_REMOTE_URL",
"")
{noformat}
Elsewhere, an env variable called REMOTE_LOAD is used to determine whether data
load scripts are loading script locally or not.
Finally, in conftest.py, there's also a flag for {{--testing_remote_cluster}}
(and a SkipIf definition based upon it). There's nothing that coordinates any
of these inputs, or guarantees that they need to agree. It's a little maddening.
> impala-shell tests break on deployed clusters if IMPALA_LOCAL_BUILD_VERSION
> is None
> ---
>
> Key: IMPALA-8552
> URL: https://issues.apache.org/jira/browse/IMPALA-8552
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Critical
>
> This is a regression introduced by the commit:
> https://github.com/apache/impala/commit/b55d905
> *Stacktrace*
> {noformat}
> shell/test_shell_commandline.py:33: in
> from util import (get_impalad_host_port, assert_var_substitution,
> run_impala_shell_cmd,
> shell/util.py:42: in
> IMPALA_HOME, "shell/build", "impala-shell-" + IMPALA_LOCAL_BUILD_VERSION,
> E TypeError: cannot concatenate 'str' and 'NoneType' objects
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8558) test_char_format failing on deployed clusters because chars_formats table does not exist
[
https://issues.apache.org/jira/browse/IMPALA-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-8558:
Description:
This issue is showing up for databases functional, functional_parquet, and
functional_orc_def. It's not immediately clear why this table wasn't created
during data load. But for example:
*Stacktrace*
{noformat}
query_test/test_chars.py:75: in test_char_format
self.run_test_case('QueryTest/chars-formats', vector)
common/impala_test_suite.py:512: in run_test_case
result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:746: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:180: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:362: in __execute_query
handle = self.execute_query_async(query_string, user=user)
beeswax/impala_beeswax.py:356: in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
beeswax/impala_beeswax.py:516: in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
EINNER EXCEPTION:
EMESSAGE: AnalysisException: Could not resolve table reference:
'chars_formats'
{noformat}
*Standard Error*
{noformat}
SET
client_identifier=query_test/test_chars.py::TestCharFormats::()::test_char_format[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|table_format:text;
-- connecting to: node-4.foo.mycompany.com:21000
-- connecting to node-4.foo.mycompany.com:21050 with impyla
-- 2019-05-15 00:05:55,759 INFO MainThread: Closing active operation
SET
client_identifier=query_test/test_chars.py::TestCharFormats::()::test_char_format[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|table_format:text;
-- executing against node-4.foo.mycompany.com:21000
use functional;
-- 2019-05-15 00:05:55,788 INFO MainThread: Started query
f845123ce97ab647:0560abac
SET
client_identifier=query_test/test_chars.py::TestCharFormats::()::test_char_format[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|table_format:text;
SET batch_size=0;
SET num_nodes=0;
SET disable_codegen_rows_threshold=0;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
-- executing against node-4.foo.mycompany.com:21000
select * from chars_formats order by vc;
{noformat}
was:
This issue is showing up for databases functional, functional_parquet, and
functional_orc_def. It's not immediately clear why this table wasn't created
during data load. But for example:
*Stacktrace*
{noformat}
query_test/test_chars.py:75: in test_char_format
self.run_test_case('QueryTest/chars-formats', vector)
common/impala_test_suite.py:512: in run_test_case
result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:746: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:180: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:362: in __execute_query
handle = self.execute_query_async(query_string, user=user)
beeswax/impala_beeswax.py:356: in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
beeswax/impala_beeswax.py:516: in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
EINNER EXCEPTION:
EMESSAGE: AnalysisException: Could not resolve table reference:
'chars_formats'
{noformat}
*Standard Error*
{noformat}
SET
client_identifier=query_test/test_chars.py::TestCharFormats::()::test_char_format[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|table_format:text;
-- connecting to: quasar-nsgjqi-4.vpc.cloudera.com:21000
-- connecting to quasar-nsgjqi-4.vpc.cloudera.com:21050 with impyla
-- 2019-05-15 00:05:55,759 INFO MainThread: Closing active operation
SET
[jira] [Created] (IMPALA-8558) test_char_format failing on deployed clusters because chars_formats table does not exist
David Knupp created IMPALA-8558:
---
Summary: test_char_format failing on deployed clusters because
chars_formats table does not exist
Key: IMPALA-8558
URL: https://issues.apache.org/jira/browse/IMPALA-8558
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.3.0
Reporter: David Knupp
This issue is showing up for databases functional, functional_parquet, and
functional_orc_def. It's not immediately clear why this table wasn't created
during data load. But for example:
*Stacktrace*
{noformat}
query_test/test_chars.py:75: in test_char_format
self.run_test_case('QueryTest/chars-formats', vector)
common/impala_test_suite.py:512: in run_test_case
result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:746: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:180: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:362: in __execute_query
handle = self.execute_query_async(query_string, user=user)
beeswax/impala_beeswax.py:356: in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
beeswax/impala_beeswax.py:516: in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
EINNER EXCEPTION:
EMESSAGE: AnalysisException: Could not resolve table reference:
'chars_formats'
{noformat}
*Standard Error*
{noformat}
SET
client_identifier=query_test/test_chars.py::TestCharFormats::()::test_char_format[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|table_format:text;
-- connecting to: quasar-nsgjqi-4.vpc.cloudera.com:21000
-- connecting to quasar-nsgjqi-4.vpc.cloudera.com:21050 with impyla
-- 2019-05-15 00:05:55,759 INFO MainThread: Closing active operation
SET
client_identifier=query_test/test_chars.py::TestCharFormats::()::test_char_format[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|table_format:text;
-- executing against quasar-nsgjqi-4.vpc.cloudera.com:21000
use functional;
-- 2019-05-15 00:05:55,788 INFO MainThread: Started query
f845123ce97ab647:0560abac
SET
client_identifier=query_test/test_chars.py::TestCharFormats::()::test_char_format[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|table_format:text;
SET batch_size=0;
SET num_nodes=0;
SET disable_codegen_rows_threshold=0;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
-- executing against quasar-nsgjqi-4.vpc.cloudera.com:21000
select * from chars_formats order by vc;
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
