[ https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16486888#comment-16486888 ]
Petr Masopust commented on LIVY-471: ------------------------------------ Using file name provided by user is bad idea. Maybe it is not so significant in Livy but I had many unreadable files just because invalid characters or simply very long file name. Storing files locally is against possible future HA Livy (see LIVY-11). If server crash (without cleaning) there is possibility to read old session's files. > New session creation API set to support resource uploading > ---------------------------------------------------------- > > Key: LIVY-471 > URL: https://issues.apache.org/jira/browse/LIVY-471 > Project: Livy > Issue Type: Improvement > Components: Server > Affects Versions: 0.5.0 > Reporter: Saisai Shao > Priority: Major > > Already post in mail list. > In our current API design to create interactive / batch session, we assume > end user should upload jars, pyFiles and related dependencies to HDFS before > creating the session, and we use one POST request to create session. But > usually end user may not have the permission to access the HDFS in their > submission machine, so it makes them hard to create new sessions. So the > requirement here is that if Livy could offer APIs to upload resources during > session creation. One implementation is proposed in > [https://github.com/apache/incubator-livy/pull/91|https://github.com/apache/incubator-livy/pull/91.] > This add a field in session creation request to delay the session creation, > then adding a bunch of APIs to support resource upload, finally adding an API > to start creating the session. This seems a feasible solution, but also a > little hack to support such scenario. So I was thinking if we could a set of > new APIs to support such scenarios, rather than hack the existing APIs. > To borrow the concept from yarn application submission, we can have 3 APIs to > create session. > * requesting a new session id from Livy Server. > * uploading resources associate with this session id. > * submitting request to create session. > This is similar to YARN's process to submit application, and we can bump the > supported API version for newly added APIs. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)