[
https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16486896#comment-16486896
]
Saisai Shao commented on LIVY-471:
----------------------------------
{quote}Using file name provided by user is bad idea. Maybe it is not so
significant in Livy but I had many unreadable files just because invalid
characters or simply very long file name.
{quote}
I don't think your proposal also solves this problem.
{quote}Storing files locally is against possible future HA Livy (see LIVY-11).
If server crash (without cleaning) there is possibility to read old session's
files.
{quote}
This should not be a problem if you really understand current Livy's session
recovery mechanism and my implementation. For the feature thing, we can plan it
in future.
Frankly, I don't want to upload resources to HDFS for Spark, because:
# Spark knows how and when to upload resources based on different cluster
manager, we don't have to make decision for Spark.
# In future if we want to support different cluster managers for Livy, like
standalone, k8s, it would be better to let Spark to decide how to handle
resources.
> New session creation API set to support resource uploading
> ----------------------------------------------------------
>
> Key: LIVY-471
> URL: https://issues.apache.org/jira/browse/LIVY-471
> Project: Livy
> Issue Type: Improvement
> Components: Server
> Affects Versions: 0.5.0
> Reporter: Saisai Shao
> Priority: Major
>
> Already post in mail list.
> In our current API design to create interactive / batch session, we assume
> end user should upload jars, pyFiles and related dependencies to HDFS before
> creating the session, and we use one POST request to create session. But
> usually end user may not have the permission to access the HDFS in their
> submission machine, so it makes them hard to create new sessions. So the
> requirement here is that if Livy could offer APIs to upload resources during
> session creation. One implementation is proposed in
> [https://github.com/apache/incubator-livy/pull/91|https://github.com/apache/incubator-livy/pull/91.]
> This add a field in session creation request to delay the session creation,
> then adding a bunch of APIs to support resource upload, finally adding an API
> to start creating the session. This seems a feasible solution, but also a
> little hack to support such scenario. So I was thinking if we could a set of
> new APIs to support such scenarios, rather than hack the existing APIs.
> To borrow the concept from yarn application submission, we can have 3 APIs to
> create session.
> * requesting a new session id from Livy Server.
> * uploading resources associate with this session id.
> * submitting request to create session.
> This is similar to YARN's process to submit application, and we can bump the
> supported API version for newly added APIs.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
