[ https://issues.apache.org/jira/browse/LIVY-782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Fogarty updated LIVY-782: -------------------------------- Summary: Idempotent Livy session creation (was: Idempotent Livy job submission) > Idempotent Livy session creation > -------------------------------- > > Key: LIVY-782 > URL: https://issues.apache.org/jira/browse/LIVY-782 > Project: Livy > Issue Type: New Feature > Components: API, Server > Reporter: Andrew Fogarty > Priority: Major > > h2. Problem description > Livy currently has POST APIs for creating sessions: > * To create a batch session, a client must submit a post request to > “/batches”. > * To create an interactive session, a client must submit a POST request to > “/sessions”. > Both APIs generate a unique session ID which is returned to the client as > part of the response payload. > These APIs are not idempotent. That is, if either the request or the > response is lost in transit, the client has no way to validate whether that > job has started. The only way to retry is to submit another POST, which > could potentially start a second job. > For example, suppose a client submits a POST to create a new batch session. > Livy receives the request and starts the batch session with ID=12. When Livy > sends the response, assume it is lost in transit due to some networking > issue. The client never receives the response, so it does not know if the > batch started correctly and does not have an ID to query the status of the > batch session. > This document contains two proposed solutions for this idempotence problem. > These solutions introduce APIs for creating sessions in an idempotent manner. > Neither solution makes changes to existing APIs. > h2. Suggested solution > This proposed solution introduces 1 new API: > * PUT(“/\{session type}/”) -> Session > This API is described below. *Note:* ‘->’ indicates the call “returns”. > h3. API: PUT(“/\{session type}/”) -> Session – Create session with given > request ID header > This new API is a PUT to create a new session (batch or interactive) for the > given session ID. This new API is very similar to the existing POST API to > create a session and expects the request payload to be a CreateBatchRequest > or CreateInteractiveRequest as appropriate. > The difference between this PUT API and the existing session POST API is that > requests to this API must contain a “requestId” header with a GUID value. If > the requestId is not provided, then PUT will fail with an error. This > requestId is saved as an optional field on the metadata object > (BatchRecoveryMetadata or InteractiveRecoveryMetadata) stored in the > SessionStore. > When creating the session, before storing the metadata object in the > SessionStore, we query the SessionStore to see if some session already exists > with that requestId. If a session with the requestId already exists, then we > return that session instead of creating a new one. If there is no existing > session with that requestId, then we create the session normally. > h3. Example > This solution solves the idempotence problem by ensuring that repeat calls to > PUT with the same requestId will return the first created session. If a > client makes a request to PUT but for some reason does not receive a > response, then they can retry that request with the same requestId. If the > session had not started, then it will start. Otherwise, if the session has > already started, then its session object will be returned to the client. > h2. Alternative solution > Introduce 2 new APIs: > # POST(“/\{session type}/id”) -> \{sessionId: Int, : GUID} > # PUT(“/\{session type}/\{session id}”) -> Session > Both are described below. > h3. API 1: POST(“/\{session type}/id”) -> \{sessionId: Int, putKey: GUID} – > Generates a new unique sessionId > The first API is a POST to generate a new unique session ID for the given > session type (batch or interactive). > This API would: > # Increment the existing sessionId incrementor. > # Store a new value \{“/putkey/\{session type}/\{session id}” -> putKey} in > the SessionStore. > # Return \{sessionId: Int, putKey: GUID} payload. > The returned payload contains the session ID as well as the “putKey”, which > is a GUID used in the second API to validate the sessionID. We call this the > “putKey” because it represents a unique key used to identify the PUT request. > We store the mapping from session ID to putKey in the SessionStore so that > the second API can validate that a provided session ID matches its putKey. > h3. API 2: PUT(“/\{session type}/\{session id}”) -> Session – Create a > session with the given session ID > The second API is a PUT to create a new session (batch or interactive) for > the given session ID. This new API is very similar to the existing POST API > to create a session and expects the request payload to be a > CreateBatchRequest or CreateInteractiveRequest as appropriate. > CreateBatchRequest and CreateInteractiveRequest will contain the optional > putKey field. > This API would: > # Validate that the provided session ID matches the putKey by reading the > \{“/putkey/\{session type}/\{session id}” value from the SessionStore. > ## If no putKey is provided, or the session ID does not match the putKey, > then we fail the request. This is to ensure that the provided sessionID was > generated by the first API, and that some client isn’t using a sessionID that > it should not have permission to use. > # Follow the usual code path to create a session, except pass down the > session ID and the putkey. > ** For this feature, we would change that code path in BatchSession and > InteractiveSession. Before saving the session metadata record to > SessionStore, we check that some record with this ID does not already exist > in the SessionStore. If it does, then we just return that session and do not > create a new session. > h3. Example > With these new APIs, a client can get a valid session ID before submitting > their batch or interactive session to Livy. > The sequence would be: > # Call POST(“/\{session type}/id”) to get a new valid session ID and putKey. > # Call PUT(“/\{session type/{session id}”) to start a new session with that > valid session ID. > # If for some reason the client does not receive a response, use the ID to > query Livy for the status. Otherwise, they can re-submit the PUT request. > When a request is re-submitted: > ## If the session had not started, it will start. > ## If the session had started already, its session object will be returned > to the client. -- This message was sent by Atlassian Jira (v8.3.4#803005)