[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches The strict hs2 protocol mode is broken when fetching large results. The FetchResults.hasMoreRows field is always returned as false. When there are no more results, Hive returns an empty batch with no rows. HIVE-26108 has been filed to support the hasMoreRows field. Added a framework test that retrieves 1M rows from tpcds. The default number of rows returned from Hive is 10K so this should be more than enough to ensure that multiple fetches are done. Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Reviewed-on: http://gerrit.cloudera.org:8080/18370 Tested-by: Impala Public Jenkins Reviewed-by: Wenzhe Zhou Reviewed-by: Csaba Ringhofer --- M shell/impala_client.py M tests/shell/test_shell_commandline.py 2 files changed, 19 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Verified Wenzhe Zhou: Looks good to me, but someone else must approve Csaba Ringhofer: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 6: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG@10 PS2, Line 10: The FetchResults.hasMoreRows field is always returned as false. When : there are no more results, Hive returns an empty batch with no rows. > Heh, I thought about this too... hasMoreRows is an optional member, so the server could decide to not set it at all. If someone sets it, then it should really mean that there are no more rows. http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py File tests/shell/test_shell_commandline.py: http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py@1100 PS2, Line 1100: > I debated with myself over the number to choose. I chose an arbitrarily hi I think it should be enough to simply set --fetch_size to a very small value - as Hive always sets hasMoreRows, we simply need more then one fetch RPCs from impala-shell (my concern is not about the size of the query from Impala perspective, but about the time needed to handle the answer in the client) -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 6 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 02 Apr 2022 08:08:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 6: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 6 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 22:04:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 6 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 22:03:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8005/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 6 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 17:30:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10381/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 14:23:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10379/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 3 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 14:11:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG@10 PS2, Line 10: The FetchResults.hasMoreRows field is always returned as false. When : there are no more results, Hive returns an empty batch with no rows. > It's not clear to me - is this a bug in Hive? Is there a ticket for it? Heh, I thought about this too... The thing is...since the thrift protocol is defined by Hive, Hive is the one that really has the final say on how things are returned. I discussed offline with a couple of people about potentially changing the server side. But right now, Hive hardcodes the "False" value into hasMoreRows. We could change that, but that has the potential (however small) of creating backward compatibility issues. Is it a bug? Well...kinda. Why have the field if you're not gonna use it? But as I mentioned, their "beeline" code clearly does not make use of this field. I just filed HIVE-26108 for this and logged it here. http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG@15 PS2, Line 15: enough to ensure that multiple fetches are done. > nit: extra line before change-id Done http://gerrit.cloudera.org:8080/#/c/18370/2/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/18370/2/shell/impala_client.py@1127 PS2, Line 1127: values > Can't this be empty if all values are null? In Impala it can't, I am not su In order for this to happen, Hive would have to be smart enough to realize that all values are null ahead of time before processing which seems very unlikely. Having said that, unlikely is not "never", so I suppose it makes sense to add this. http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py File tests/shell/test_shell_commandline.py: http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py@1100 PS2, Line 1100: 100 > Do we need to run such a large query? Wouldn't it be enough to have more I debated with myself over the number to choose. I chose an arbitrarily high number because the default can theoretically change and I didn't want to lose this test if that happened. Having said that, I'm open to lowering it a little bit, I'll change it to 100,000 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 14:04:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Hello Aman Sinha, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18370 to look at the new patch set (#5). Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches The strict hs2 protocol mode is broken when fetching large results. The FetchResults.hasMoreRows field is always returned as false. When there are no more results, Hive returns an empty batch with no rows. HIVE-26108 has been filed to support the hasMoreRows field. Added a framework test that retrieves 1M rows from tpcds. The default number of rows returned from Hive is 10K so this should be more than enough to ensure that multiple fetches are done. Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 --- M shell/impala_client.py M tests/shell/test_shell_commandline.py 2 files changed, 19 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/5 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Hello Aman Sinha, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18370 to look at the new patch set (#4). Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches The strict hs2 protocol mode is broken when fetching large results. The FetchResults.hasMoreRows field is always returned as false. When there are no more results, Hive returns an empty batch with no rows. HIVE-26108 has been filed to support the hasMoreRows field. Added a framework test that retrieves 1M rows from tpcds. The default number of rows returned from Hive is 10K so this should be more than enough to ensure that multiple fetches are done. Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 --- M shell/impala_client.py M tests/shell/test_shell_commandline.py 2 files changed, 19 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/4 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 4 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Hello Aman Sinha, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18370 to look at the new patch set (#3). Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches The strict hs2 protocol mode is broken when fetching large results. The FetchResults.hasMoreRows field is always returned as false. When there are no more results, Hive returns an empty batch with no rows. Added a framework test that retrieves 1M rows from tpcds. The default number of rows returned from Hive is 10K so this should be more than enough to ensure that multiple fetches are done. Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 --- M shell/impala_client.py M tests/shell/test_shell_commandline.py 2 files changed, 19 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/3 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 3 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG@10 PS2, Line 10: The FetchResults.hasMoreRows field is always returned as false. When : there are no more results, Hive returns an empty batch with no rows. It's not clear to me - is this a bug in Hive? Is there a ticket for it? http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG@15 PS2, Line 15: enough to ensure that multiple fetches are done. nit: extra line before change-id http://gerrit.cloudera.org:8080/#/c/18370/2/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/18370/2/shell/impala_client.py@1127 PS2, Line 1127: values Can't this be empty if all values are null? In Impala it can't, I am not sure about Hive. It may make sense to add a test for this (setting fetch size to 1 and run a query with multiple lines where the first line is always NULL http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py File tests/shell/test_shell_commandline.py: http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py@1100 PS2, Line 1100: 100 Do we need to run such a large query? Wouldn't it be enough to have more rows than fetch size? -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 12:07:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 09:32:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8002/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 05:02:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8000/ -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 04:53:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 2: Code-Review+2 (1 comment) Carry Wenzhe's +1 and add mine and bumping to +2 http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py@1126 PS1, Line 1126: tcol = col_value_converters[0][0](resp.results.columns[0]) > Yeah, this is the only way as far as I can tell. In that case, this should be good. -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 00:30:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8000/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 00:23:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 00:21:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/18370/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18370/1//COMMIT_MSG@12 PS1, Line 12: > nit: add a sentence about what type of test has been added. Done http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py@1126 PS1, Line 1126: tcol = col_value_converters[0][0](resp.results.columns[0]) > Is checking the number of columns the safest way ? Is there an alternative Yeah, this is the only way as far as I can tell. I looked at Hive Beeline implementation and they iterate through the columns. There is no field which provides the number of rows. The columns are the only place where we can find this information. -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 31 Mar 2022 21:30:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Hello Aman Sinha, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18370 to look at the new patch set (#2). Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches The strict hs2 protocol mode is broken when fetching large results. The FetchResults.hasMoreRows field is always returned as false. When there are no more results, Hive returns an empty batch with no rows. Added a framework test that retrieves 1M rows from tpcds. The default number of rows returned from Hive is 10K so this should be more than enough to ensure that multiple fetches are done. Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 --- M shell/impala_client.py M tests/shell/test_shell_commandline.py 2 files changed, 13 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/2 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/18370/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18370/1//COMMIT_MSG@12 PS1, Line 12: nit: add a sentence about what type of test has been added. http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py@1126 PS1, Line 1126: tcol = col_value_converters[0][0](resp.results.columns[0]) Is checking the number of columns the safest way ? Is there an alternative to simply check the number of rows such that we don't have to dereference a column array ? Or perhaps if this pattern is used elsewhere here let me know. -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 31 Mar 2022 18:19:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7986/ -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 31 Mar 2022 03:49:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7986/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Mar 2022 23:21:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10366/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Mar 2022 23:18:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Steve Carlin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18370 Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches The strict hs2 protocol mode is broken when fetching large results. The FetchResults.hasMoreRows field is always returned as false. When there are no more results, Hive returns an empty batch with no rows. Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 --- M shell/impala_client.py M tests/shell/test_shell_commandline.py 2 files changed, 13 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/1 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin
