Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/10939 )
Change subject: IMPALA-7259: Improve Impala shell performance ...................................................................... IMPALA-7259: Improve Impala shell performance This patch fixes the slow performance in Impala shell, especially for large queries by replacing all calls to sqlparse.format(sql_string, strip_comments=True) with the custom implementation of strip comments that does not use grouping. The code to strip leading comments was also refactored to not use grouping. * Benchmark running a query with 12K columns * Before the patch: $ time impala-shell.sh -f large.sql --quiet real 2m4.154s user 2m0.536s sys 0m0.088s After the patch: $ time impala-shell.sh -f large.sql --quiet real 0m3.885s user 0m1.516s sys 0m0.048s Testing: - Added a new test to test the Impala shell performance - Ran all shell tests on Python 2.6 and Python 2.7 Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 --- M shell/impala_shell.py A tests/shell/test_file_large.sql M tests/shell/test_shell_commandline.py 3 files changed, 16,051 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/10939/8 -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya <fwij...@cloudera.com> Gerrit-Reviewer: Fredy Wijaya <fwij...@cloudera.com> Gerrit-Reviewer: Michael Brown <mi...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
