[
https://issues.apache.org/jira/browse/IMPALA-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541299#comment-16541299
]
Nghia Le commented on IMPALA-7259:
----------------------------------
I have done some experiments using cProfile with your file and found out that
there are 2 main functions which made the shell slow with large queries:
parse_query_text() and strip_leading_comment() in impala_shell.py
For parse_query_text():
The problem is in this statement
``` if query_list and not sqlparse.format(query_list[-1],
strip_comments=True).strip("\n"):
query_list.pop()
Since your last query is a really long string, the sqlparse.format needs a
lot of time to check whether that string is a comment or not.
For strip_leading_comment():
The problem also lies at string processing part.
> impala-shell is weirdly slow with some large queries
> ----------------------------------------------------
>
> Key: IMPALA-7259
> URL: https://issues.apache.org/jira/browse/IMPALA-7259
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
> Affects Versions: Impala 3.1.0
> Reporter: Tim Armstrong
> Assignee: Nghia Le
> Priority: Major
> Attachments: wide-parquet-agg.sql
>
>
> impala-shell is very slow at processing some large queries - it takes over a
> minute to actually submit the query. I've attached an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
