[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 11 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 05:56:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. When result spooling is disabled, the timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. When result spooling is enabled, a client can fetch multiple RowBatches at a time, so the timeout controls the total time spent waiting for RowBatches to be produced. The timeout applies to both waiting for rows to be sent by the fragment instance thread, and waiting for rows to be materialized (e.g. the time measured by RowMaterializationTimer). Testing: * Added new tests to test_fetch.py * Ran core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Reviewed-on: http://gerrit.cloudera.org:8080/14157 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/exprs/expr-test.cc M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 14 files changed, 196 insertions(+), 9 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 12 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 11 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 00:56:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4916/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 11 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 00:56:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 10: The "first row fetched" behaviour seems like an improvement, since previously it might report the first fetch too early anyway. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 10 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 00:46:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4504/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 10 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 09 Sep 2019 17:49:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4503/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 9 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 09 Sep 2019 17:46:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/14157/9/tests/hs2/test_fetch.py File tests/hs2/test_fetch.py: http://gerrit.cloudera.org:8080/#/c/14157/9/tests/hs2/test_fetch.py@372 PS9, Line 372: ) > flake8: E501 line too long (91 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 9 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 09 Sep 2019 17:13:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 9: @Tim, think I might need another review on this one. The test failures in the pre-commit job were legit. The fix is pretty simple, see the change to expr-test.cc. I noticed a few other odd things while testing this though, so had to make the following changes: * Before the latest patch, if result spooling was disabled, the timeout applied to both the time taking for the fragment thread to send rows AND the time taken to materialize rows. If result spooling was disabled, it was only applied to the time taken for the fragment thread to send rows. The latest patch fixes that, and I added another test for it as well. * I started seeing duplicate entries for "First row fetched" in the profile; it looks like the current code emits "First row fetched" whenever ImpalaServer::FetchInternal is called and 0 rows have been fetched. The issue is that with the timeout, it is not possible for a fetch request to return 0 rows, and for more rows to be available. Fixed this by moving the "First row fetched" event to "Coordinator::GetNext". -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 9 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 09 Sep 2019 17:12:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14157 to look at the new patch set (#10). Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. When result spooling is disabled, the timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. When result spooling is enabled, a client can fetch multiple RowBatches at a time, so the timeout controls the total time spent waiting for RowBatches to be produced. The timeout applies to both waiting for rows to be sent by the fragment instance thread, and waiting for rows to be materialized (e.g. the time measured by RowMaterializationTimer). Testing: * Added new tests to test_fetch.py * Ran core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/exprs/expr-test.cc M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 14 files changed, 196 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/10 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 10 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/14157/9/tests/hs2/test_fetch.py File tests/hs2/test_fetch.py: http://gerrit.cloudera.org:8080/#/c/14157/9/tests/hs2/test_fetch.py@372 PS9, Line 372: ) flake8: E501 line too long (91 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 9 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 09 Sep 2019 17:06:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14157 to look at the new patch set (#9). Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. When result spooling is disabled, the timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. When result spooling is enabled, a client can fetch multiple RowBatches at a time, so the timeout controls the total time spent waiting for RowBatches to be produced. The timeout applies to both waiting for rows to be sent by the fragment instance thread, and waiting for rows to be materialized (e.g. the time measured by RowMaterializationTimer). Testing: * Added new tests to test_fetch.py * Ran core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/exprs/expr-test.cc M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 14 files changed, 195 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/9 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 9 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4893/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Sep 2019 13:42:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Sep 2019 13:42:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4471/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Sep 2019 18:35:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Sep 2019 18:30:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4472/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Sep 2019 18:24:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14157 to look at the new patch set (#7). Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. When result spooling is disabled, the timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. When result spooling is enabled, a client can fetch multiple RowBatches at a time, so the timeout controls the total time spent waiting for RowBatches to be produced. Testing: * Added new tests to test_fetch.py * Running core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 9 files changed, 159 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/7 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/14157/5/be/src/exec/blocking-plan-root-sink.cc File be/src/exec/blocking-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/5/be/src/exec/blocking-plan-root-sink.cc@124 PS5, Line 124: bool timed_out = > Might be simpler to initialise to false. Done http://gerrit.cloudera.org:8080/#/c/14157/5/be/src/exec/blocking-plan-root-sink.cc@137 PS5, Line 137: (uint64_t) 1 > nit: shouldn't use c-style cast Done http://gerrit.cloudera.org:8080/#/c/14157/5/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/5/be/src/exec/buffered-plan-root-sink.cc@163 PS5, Line 163: PlanRootSink::fetch_rows_timeout_us() <= wait_timeout_timer.ElapsedTime(); > Might be simpler to initialise to false. I think we probably want to try at Done -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Sep 2019 17:54:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14157 to look at the new patch set (#6). Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. When result spooling is disabled, the timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. When result spooling is enabled, a client can fetch multiple RowBatches at a time, so the timeout controls the total time spent waiting for RowBatches to be produced. Testing: * Added new tests to test_fetch.py * Running core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 9 files changed, 159 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/6 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/14157/5/be/src/exec/blocking-plan-root-sink.cc File be/src/exec/blocking-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/5/be/src/exec/blocking-plan-root-sink.cc@124 PS5, Line 124: bool timed_out = Might be simpler to initialise to false. http://gerrit.cloudera.org:8080/#/c/14157/5/be/src/exec/blocking-plan-root-sink.cc@137 PS5, Line 137: (uint64_t) 1 nit: shouldn't use c-style cast http://gerrit.cloudera.org:8080/#/c/14157/5/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/5/be/src/exec/buffered-plan-root-sink.cc@163 PS5, Line 163: PlanRootSink::fetch_rows_timeout_us() <= wait_timeout_timer.ElapsedTime(); Might be simpler to initialise to false. I think we probably want to try at least once to get rows otherwise things could (potentially) end up livelocked in some extreme cases. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Sep 2019 16:17:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4464/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Sep 2019 00:29:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@159 PS4, Line 159: > Don't need _ suffix on local variable. Done http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@172 PS4, Line 172: // wait_timeout_timer ensures the client only ever waits up to > Why not just start the timer and leave it running? I think the elapsed time Yeah makes sense. Moved the Start before the lock is acquired, and it is never stopped. This way it tracks the total time required to fetch, including waiting for rows to become available and waiting for rows to materialize. I have some additional tests in mind for this, but they require IMPALA-8825 first. http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@174 PS4, Line 174: uint64_t wait_duration = max((uint64_t) 1, > nit: could be a one liner: Done -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 04 Sep 2019 23:54:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14157 to look at the new patch set (#5). Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. When result spooling is disabled, the timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. When result spooling is enabled, a client can fetch multiple RowBatches at a time, so the timeout controls the total time spent waiting for RowBatches to be produced. Testing: * Added new tests to test_fetch.py * Running core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 9 files changed, 161 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/5 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@159 PS4, Line 159: wait_timeout_timer_ Don't need _ suffix on local variable. http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@172 PS4, Line 172: wait_timeout_timer_.Start(); Why not just start the timer and leave it running? I think the elapsed time is likely more important to the client than the time spent blocked. http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@174 PS4, Line 174: if (!rows_available_.WaitFor(l, wait_duration)) { nit: could be a one liner: timed_out = !rows_available_.WaitFor(...); -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 04 Sep 2019 19:08:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4449/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Sep 2019 22:15:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/exec/blocking-plan-root-sink.cc File be/src/exec/blocking-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/exec/blocking-plan-root-sink.cc@126 PS3, Line 126: timed_out = true; > At this point is it possible that results had some rows appended to it? I t Done http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/exec/buffered-plan-root-sink.cc@169 PS3, Line 169: if (!rows_available_.WaitFor(l, PlanRootSink::fetch_rows_timeout_us())) { > Intuitively I would have expected the timeout to apply to the cumulative ti Done http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/service/query-options.h File be/src/service/query-options.h: http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/service/query-options.h@189 PS3, Line 189: TQueryOptionLevel::DEVELOPMENT) > I think this one should be ok to expose immediately, right, since it's bein Done -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Sep 2019 21:35:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14157 to look at the new patch set (#4). Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. When result spooling is disabled, the timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. When result spooling is enabled, a client can fetch multiple RowBatches at a time, so the timeout controls the total time spent waiting for RowBatches to be produced. Testing: * Added new tests to test_fetch.py * Running core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 9 files changed, 154 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/4 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 3: (3 comments) Cool! I didn't expect to see this so soon. http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/exec/blocking-plan-root-sink.cc File be/src/exec/blocking-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/exec/blocking-plan-root-sink.cc@126 PS3, Line 126: timed_out = true; At this point is it possible that results had some rows appended to it? I think it's at very least subtle whether the timeout can expire and WaitFor() return false in the case when the producer appended and signalled. Might be good to comment on whether this code path needs to handle that case (it looks like it handles it correctly already). http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/exec/buffered-plan-root-sink.cc@169 PS3, Line 169: if (!rows_available_.WaitFor(l, PlanRootSink::fetch_rows_timeout_us())) { Intuitively I would have expected the timeout to apply to the cumulative time spent blocked here - the RPC could still block for a long time if data was trickling in within the timeout period. http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/service/query-options.h File be/src/service/query-options.h: http://gerrit.cloudera.org:8080/#/c/14157/3/be/src/service/query-options.h@189 PS3, Line 189: TQueryOptionLevel::DEVELOPMENT) I think this one should be ok to expose immediately, right, since it's being used in the non-spooling path - it could be advanced? -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Aug 2019 02:23:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4426/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Aug 2019 01:00:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14157 to look at the new patch set (#3). Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. The timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. Since result spooling allows clients to fetch multiple RowBatches at a time, the timeout is reset when each RowBatch is read by the client thread. Testing: * Added new tests to test_fetch.py * Running core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 9 files changed, 111 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/3 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4422/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 28 Aug 2019 20:58:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4421/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 28 Aug 2019 20:55:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/14157/1/be/src/exec/blocking-plan-root-sink.cc File be/src/exec/blocking-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/1/be/src/exec/blocking-plan-root-sink.cc@117 PS1, Line 117: // True if the consumer timed out waiting for the producer to send rows, false otherwise. > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/14157/1/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/1/be/src/exec/buffered-plan-root-sink.cc@155 PS1, Line 155: // True if the consumer timed out waiting for the producer to send rows, false otherwise. > line too long (93 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 28 Aug 2019 20:17:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14157 to look at the new patch set (#2). Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. The timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. Since result spooling allows clients to fetch multiple RowBatches at a time, the timeout is reset when each RowBatch is read by the client thread. Testing: * Added new tests to test_fetch.py * Running core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 9 files changed, 111 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/2 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/14157/1/be/src/exec/blocking-plan-root-sink.cc File be/src/exec/blocking-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/1/be/src/exec/blocking-plan-root-sink.cc@117 PS1, Line 117: // True if the consumer timed out waiting for the producer to send rows, false otherwise. line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/14157/1/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/1/be/src/exec/buffered-plan-root-sink.cc@155 PS1, Line 155: // True if the consumer timed out waiting for the producer to send rows, false otherwise. line too long (93 > 90) -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Aug 2019 20:15:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14157 Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. The timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. Since result spooling allows clients to fetch multiple RowBatches at a time, the timeout is reset when each RowBatch is read by the client thread. Testing: * Added new tests to test_fetch.py * Running core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 9 files changed, 109 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/1 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar