[
https://issues.apache.org/jira/browse/LENS-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526179#comment-15526179
]
Rajat Khandelwal commented on LENS-743:
---------------------------------------
h4. Query State added: FAILING.
To start with, the FAILED state is renamed to FAILING. From failing, query
immediately moves to either FAILED or QUEUED depending on whether retries are
happening or not. So, FAILING is an intermittent state and no events are fired
for this. But, when the query moves from FAILING to QUEUED, a different event
is fired than when the query moves from NEW to QUEUED. So event listeners can
listen to that event to know whether the query is being attempted for the first
time or not. The new event class is in very basic shape and can evolve as and
when event listeners are required.
h4. Query Constraints can now return String
Query constraint classes were returning boolean indicating whether the query is
allowed to launch or not. Now they are returning String. The return value
should be null if the query is allowed to launch and a String denoting reason
of rejection in case the query isn't allowed to launch. This message is then
further used as a progress message of the query. This will improve users'
knowledge of what's happening to their queries.
h4. Query Comparator design change
The earlier design was subclass based. a comparator can extend another
comparator and can call super.compare in case its own comparison logic return
0. New design is chain based. There is a ChainedComparator which holds a list
of comparators, For comparing two objects, each of the comparators in the chain
are asked to compare the objects until one of them returns non-zero value.
In line with this, some of the classes are moved to another package.
h4. Status Update Retry classes generified
Have generified classes used for status update retries, so that the common
logic can be used for query retries too. In line with this, have changed
package names of a bunch of classes.
h4. Driver API change
Driver now has
{code}
RetryPolicyDecider<QueryContext> getRetryPolicyDecider();
{code}
h4. Query Context
Query context now holds a driver retry policy and a server retry policy. On
failure, driver retry policy is consulted as to whetner retries are exhausted.
The driver retry policy stays the same as long as the query is being
run/retried on the same driver. Once a driver gives up on a query, a server
retry policy is decided for the query. The old driver is removed as candidates
and a new driver is selected. If there are no drivers available, the query is
marked failed with the old driver. If a driver _is_ selected, query is retried
on that driver. In case of failures there, the new driver's retry policy will
be constructed and consulted.
Other than the policies, there is a List of failed attempts added to the Query
Context. Each time retry is planned, the current state is extracted out as a
failed attempt.
Both the failed attempts and policies are persisted and recovered on restart.
h4. Retry policies
Retry policies are picked by reflection. Class name is to be provided in lens
config and driver configs.
h4. Backoffs
Retry policies can ask for immediate/ASAP retry, or ask for backoff. So on
failure if retries are to be done, the query is moved from FAILING to QUEUED.
In QuerySubmitter, the effective retry policy is asked to determine whether to
retry immediately or after some backoff. If there's backoff, the query is moved
to waiting, and a timed task is scheduled which will move the query from
waiting to queued.
h4. Test Cases
Added test cases for:
* Retry on same driver
* Failure, selection of different driver and succeeding on new driver
* Failure, no candidate drivers available to retry => Failure
* Backoffs verification between launches
* Restart doesn't affect number of retries and backoffs.
> Query failure retries for transient errors
> ------------------------------------------
>
> Key: LENS-743
> URL: https://issues.apache.org/jira/browse/LENS-743
> Project: Apache Lens
> Issue Type: Improvement
> Components: server
> Reporter: Amareshwari Sriramadasu
> Assignee: Rajat Khandelwal
> Labels: gsoc2016, java
>
> There have to be retries for query failures for transient errors like network
> errors (Hive server not reachable/ Metastore not reachable/ DB not
> reachable). Retries should be available for each phase - submission,
> execution, updating status, fetching results and formatting.
> Right now, any such failure results in marking query as failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
