Hello Thomas Tauber-Marshall, Sahil Takiar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16323
to look at the new patch set (#3).
Change subject: IMPALA-9225: Add query option for retryable queries to spool
all results before returning any to the client
......................................................................
IMPALA-9225: Add query option for retryable queries to spool all results before
returning any to the client
If we have returned any results to the client in the original query,
query retry will be skipped to avoid incorrect results. This patch adds
a query option, safely_retry_queries, for retryable queries to spool all
results before returning any to the client. It defaults to true. If all
query results cannot be contained in the allocated result spooling
space, we'll return results and thus disabled query retry on the query.
Setting safely_retry_queries to false will fallback to the original
behavior - client can fetch results when any of them are ready. So we
explicitly set it to false in the retried query since it won't be
retried anymore. For non retryable queries, the safely_retry_queries
option takes no effects.
To implement this, this patch defers the time when a retryable query
runs into the FINISHED state (rows available). If safely_retry_queries
is true, results spooling will be implicitly enabled for a retryable
query. The coordinator fragment instance will only update its
opened_promise_ when the sender is blocked by a full queue or all
results are spooled, or when any errors happen. The DataSink::Send()
interface is extended to pass in a reference of the promise. So the sink
can update it to signal the fetch() request which is waiting for the
opened_promise_ to continue and actually fetch results.
Tests:
- Add a test to verify that a retryable query will spool all its results
when safely_retry_queries is true (no matter whether
spool_query_results is true or false).
- Add a test to verify that query retry succeeds when a retryable query
is spooling its results (safely_retry_queries=true).
- Add a test to verify that the retried query won't implicitly enable
result spooling.
- Add a test to verify that the original query can be canceled
correctly. We need this because the added logics for
safely_retry_queries are related to a condition variable that could
affect the cancellation code path.
- Add a test to verify results will be returned when all of them can't
fit into the result spooling space, and query retry will be skipped.
- Run CORE tests.
Change-Id: I462dbfef9ddab9060b30a6937fca9122484a24a5
---
M be/src/exec/blocking-plan-root-sink.cc
M be/src/exec/blocking-plan-root-sink.h
M be/src/exec/buffered-plan-root-sink.cc
M be/src/exec/buffered-plan-root-sink.h
M be/src/exec/data-sink.h
M be/src/exec/hbase-table-sink.cc
M be/src/exec/hbase-table-sink.h
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/exec/kudu-table-sink.cc
M be/src/exec/kudu-table-sink.h
M be/src/exec/nested-loop-join-builder.cc
M be/src/exec/nested-loop-join-builder.h
M be/src/exec/partitioned-hash-join-builder.cc
M be/src/exec/partitioned-hash-join-builder.h
M be/src/exec/plan-root-sink.cc
M be/src/exec/plan-root-sink.h
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
M be/src/runtime/query-driver.cc
M be/src/runtime/spillable-row-batch-queue.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_query_retries.py
29 files changed, 291 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/16323/3
--
To view, visit http://gerrit.cloudera.org:8080/16323
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I462dbfef9ddab9060b30a6937fca9122484a24a5
Gerrit-Change-Number: 16323
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tmarsh...@cloudera.com>
