[ https://issues.apache.org/jira/browse/LIVY-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Saisai Shao resolved LIVY-735. ------------------------------ Fix Version/s: 0.8.0 Assignee: runzhiwang Resolution: Fixed Issue resolved by pull request 268 https://github.com/apache/incubator-livy/pull/268 > Fix RPC Channel Closed When Multi Clients Connect to One Driver > ---------------------------------------------------------------- > > Key: LIVY-735 > URL: https://issues.apache.org/jira/browse/LIVY-735 > Project: Livy > Issue Type: Sub-task > Reporter: Yiheng Wang > Assignee: runzhiwang > Priority: Major > Fix For: 0.8.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Currently, the driver tries to support communicating with multi-clients, by > registering each client at > https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java#L220. > But actually, if multi-clients connect to one driver, the rpc channel will > close, the reason are as follows. > 1. In every communication, client sends two packages to driver: header\{type, > id}, and payload at > https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/rpc/RpcDispatcher.java#L144. > 2. If client1 sends header1, payload1, and client2 sends header2, payload2 at > the same time. > The driver receives the package in the order: header1, header2, payload1, > payload2. > 3. When driver receives header1, driver assigns lastHeader at > https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/rpc/RpcDispatcher.java#L73. > 4. Then driver receives header2, driver process it as a payload at > https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/rpc/RpcDispatcher.java#L78 > which cause exception and rpc channel closed. > In the muti-active HA mode, the design doc is at: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing, > the session is allocated among servers by consistent hashing. If a new livy > joins, some session will be migrated from old livy to new livy. If the > session client in new livy connect to driver before stoping session client in > old livy, then two session clients will both connect to driver, and rpc > channel close. In this case, it's hard to ensure only one client connect to > one driver at any time. So it's better to support multi-clients connect to > one driver, which has no side effects. > How to fix: > 1. Move the code of processing client message from `RpcDispatcher` to each > `Rpc`. > 2. Each `Rpc` registers itself to `channelRpc` in RpcDispatcher. > 3. `RpcDispatcher` dispatches each message to `Rpc` according to > `ctx.channel()`. -- This message was sent by Atlassian Jira (v8.3.4#803005)