[
https://issues.apache.org/jira/browse/IMPALA-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097912#comment-18097912
]
ASF subversion and git services commented on IMPALA-15121:
----------------------------------------------------------
Commit 450aad373856167c18136adce92749550a979a71 in impala's branch
refs/heads/master from Yida Wu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=450aad373 ]
IMPALA-15121: Fix query failure when catalogd is in standby mode during HA
failover
During a catalogd HA failover, an impalad coordinator might try
to fetch metadata from a catalogd that was previously in active
mode but has just restarted and in standby mode now. In this
case, the metadata request is rejected with a standby mode
error, causing the query to fail.
This change recognizes the standby mode rejection error and
converts it into an InconsistentMetadataFetchException, allowing
the existing Frontend retry mechanism to retry the metadata
fetch. As a result, the query retries against the new active
catalogd at the end instead of failing.
Added thrift error code CATALOG_IN_STANDBY_MODE for rejection
on standby mode.
Tests:
Added a new ee test test_standby_retry_during_failover.
Passed core tests.
Change-Id: Ia511e22a596733e6a2dd752894d6da43e91faca6
Reviewed-on: http://gerrit.cloudera.org:8080/24600
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Queries fail with "catalogd is in standby mode" during HA failover
> ------------------------------------------------------------------
>
> Key: IMPALA-15121
> URL: https://issues.apache.org/jira/browse/IMPALA-15121
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 4.5.0
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> When Catalogd HA is enabled, queries can fail during a catalogd failover with
> the following error:
> {code:java}
> java.lang.RuntimeException:
> org.apache.impala.catalog.local.LocalCatalogException: Unable to load
> database names
> CAUSED BY: TException: Request for Catalog service is rejected since catalogd
> <host>:<port> is in standby mode
> {code}
> Reproduce steps:
> 1. Kill the active catalogd.
> 2. Run a query that needs to access the catalog:
> {code:java}
> refresh functional.alltypes;
> {code}
> 3. Restart the killed catalogd.
> The root cause is that when the catalogd is down, the frontend of impalad
> will keep trying to do RPCs with the current active catalogd, until it
> eventually comes up again. Because it comes back in standby status, the
> frontend will receive the standby rejection error introduced in IMPALA-12849,
> which ends up as a query failure. We should retry when receiving this error
> on the impalad side so that it can wait to receive the statestore update and
> talk to the correct active catalogd without a query failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
