[
https://issues.apache.org/jira/browse/LIVY-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiheng Wang updated LIVY-696:
-----------------------------
Description:
This JIRA is to discuss support recovery in Livy Thrift server.
Livy server support recovery. If user set *livy.server.recovery.mode* to
*recovery* and set the *state-store*, livy server will persist its states. Once
livy server crashes, the sessions will be restored after the service restart.
The recovery is not yet supported in thrift server. After the service restarts,
all JDBC connections and statements will be lost.
Should we support recovery in the thrift server? There's one concern. In JDBC
binary mode, the connection will be lost if the server crashes, and looks like
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC
http mode, we will benefit from the recovery. As http mode use short http
connections instead of long connection.
There're also two levels of recovery need to be considered.
*Session recovery* let user can still use their JDBC connection after service
restart. It should be straight forward to implement. We just need to persist
the JDBC session/livy session mapping to the state-store and recover the
mapping when restarting.
*Statement recovery* let user can still use the JDBC statement(or fetch result)
after service restart. This needs to persist ExecuteStatementOperation state.
The concern is some states are not suitable to persist to state-store, e.g.
operationMessages, rowOffset, state, operationException, backgroundHandle,
lastAccessTime, operationComplete. They're complex type or change frequently.
was:
This JIRA is to discuss support recovery in Livy Thrift server.
Livy server support recovery. If user set *livy.server.recovery.mode* to
*recovery* and set the *state-store*, livy server will persist its states. Once
livy server crashes, the sessions will be restored after the service restart.
The recovery is not yet supported in thrift server. After the service restarts,
all JDBC connections and statements will be lost.
Should we support recovery in the thrift server? There's one concern. In JDBC
binary mode, the connection will be lost if the server crashes, and looks like
hive-jdbc cannot reuse session-id when reconnecting to the server. But in JDBC
http mode, we will benefit from the recovery. As http mode use short http
connections instead of long connection.
There're also two levels of recovery need to be considered.
*Session recovery* let user can still use their JDBC connection after service
restart. It should be straight forward to implement. We just need to persist
the JDBC session/livy session mapping to the state-store and recover the
mapping when restarting.
*Operation recovery* let user can still use the JDBC statement(or fetch result)
after service restart. This needs to persist ExecuteStatementOperation state.
The concern is some states are not suitable to persist to state-store, e.g.
operationMessages, rowOffset, state, operationException, backgroundHandle,
lastAccessTime, operationComplete. They're complex type or change frequently.
> Support recovery in livy thrift server
> --------------------------------------
>
> Key: LIVY-696
> URL: https://issues.apache.org/jira/browse/LIVY-696
> Project: Livy
> Issue Type: Improvement
> Components: Thriftserver
> Reporter: Yiheng Wang
> Priority: Major
>
> This JIRA is to discuss support recovery in Livy Thrift server.
> Livy server support recovery. If user set *livy.server.recovery.mode* to
> *recovery* and set the *state-store*, livy server will persist its states.
> Once livy server crashes, the sessions will be restored after the service
> restart.
> The recovery is not yet supported in thrift server. After the service
> restarts, all JDBC connections and statements will be lost.
> Should we support recovery in the thrift server? There's one concern. In JDBC
> binary mode, the connection will be lost if the server crashes, and looks
> like hive-jdbc cannot reuse session-id when reconnecting to the server. But
> in JDBC http mode, we will benefit from the recovery. As http mode use short
> http connections instead of long connection.
> There're also two levels of recovery need to be considered.
> *Session recovery* let user can still use their JDBC connection after service
> restart. It should be straight forward to implement. We just need to persist
> the JDBC session/livy session mapping to the state-store and recover the
> mapping when restarting.
> *Statement recovery* let user can still use the JDBC statement(or fetch
> result) after service restart. This needs to persist
> ExecuteStatementOperation state. The concern is some states are not suitable
> to persist to state-store, e.g. operationMessages, rowOffset, state,
> operationException, backgroundHandle, lastAccessTime, operationComplete.
> They're complex type or change frequently.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
