Re: user defined sessionId / URI for Livy sessions

2017-09-13 Thread Meisam Fathi 
>
>
> E.g. something like this may be useful in an active-active livy
> configuration because its not clear how sequential numeric id's would work
> in that context. Perhaps UUID's would also suffice for the active-active
> setup.
>

For active-active, sequential numeric IDs could be generated by
`StateStore`/`SessionStore` to guarantee uniqueness. I see how having UUIDs
can be a different way of solving the problem.

Thanks,
Meisam

Re: user defined sessionId / URI for Livy sessions

2017-09-12 Thread Bikas Saha 
Resending to both aliases

Hi,

Would be good to understand how this feature would be useful other than 
providing users a handle that they can rely on as a 3rd party.

E.g. something like this may be useful in an active-active livy configuration 
because its not clear how sequential numeric id's would work in that context. 
Perhaps UUID's would also suffice for the active-active setup. Maybe the use 
case for user defined sessions could be to integrate with other software that 
could reliably use the user defined name for its internal operations instead of 
using a livy session id thats not guaranteed to be unique.

Curious as to what motivated choosing sequential numbers for livy sessions?

Bikas



From: Alex Bozarth <ajboz...@us.ibm.com>
Sent: Tuesday, September 12, 2017 2:44 PM
To: user@livy.incubator.apache.org
Cc: d...@livy.incubator.apache.org
Subject: Re: user defined sessionId / URI for Livy sessions


That's fair, I'll still withhold judgement until I see an impl though.



Alex Bozarth
Software Engineer
Spark Technology Center


E-mail: ajboz...@us.ibm.com<mailto:ajboz...@us.ibm.com>
GitHub: github.com/ajbozarth<https://github.com/ajbozarth>


505 Howard Street
San Francisco, CA 94105
United States




[Inactive hide details for Marcelo Vanzin ---09/12/2017 02:40:33 PM---BTW take 
my comment there with a grain of salt; at the tim]Marcelo Vanzin ---09/12/2017 
02:40:33 PM---BTW take my comment there with a grain of salt; at the time Livy 
was being targeted at mostly being

From: Marcelo Vanzin <van...@cloudera.com>
To: d...@livy.incubator.apache.org
Cc: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>
Date: 09/12/2017 02:40 PM
Subject: Re: user defined sessionId / URI for Livy sessions





BTW take my comment there with a grain of salt; at the time Livy was being 
targeted at mostly being hidden from users, making such a feature not make much 
sense in Livy itself. But things may have changed since then, especially as 
people started using it more.

On Mon, Sep 11, 2017 at 1:51 PM, Alex Bozarth 
<ajboz...@us.ibm.com<mailto:ajboz...@us.ibm.com>> wrote:

I would agree with Marcelo's comment the JIRA that this isn't a good feature 
for livy, but I'll take a look at your impl if you open a PR and see if it 
changes my mind.


Alex Bozarth
Software Engineer
Spark Technology Center


E-mail: ajboz...@us.ibm.com<mailto:ajboz...@us.ibm.com>
GitHub: 
github.com/ajbozarth<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ajbozarth=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=VdRly7JyucmN1BnLWxZGFJY-iemUUfqUWOlQqaGAA54=047UIBZ1bxlXg56NfOnZ2UsJMzTJTSJ34UGFbLo3GbA=>


505 Howard Street
San Francisco, CA 94105
United States




Meisam Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in adding 
this feature to Livy? I can send a PR

From: Meisam Fathi <meisam.fa...@gmail.com<mailto:meisam.fa...@gmail.com>>
To: "user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>" 
<user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>>, 
"d...@livy.incubator.apache.org<mailto:d...@livy.incubator.apache.org>" 
<d...@livy.incubator.apache.org<mailto:d...@livy.incubator.apache.org>>
Date: 09/11/2017 10:23 AM
Subject: Re: user defined sessionId / URI for Livy sessions




+ dev
Is there any interest in adding this feature to Livy? I can send a PR

Ideally, it would be helpful if we could mint a session ID with a PUT
> request, something like PUT /sessions/foobar, where "foobar" is the newly
> created sessionId.
>
> I suggest we make session names unique and nonnumeric values (to guarantee
a session name does not clash with another session name or session ID).

Design doc:
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=
JIRA ticket: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=


Thanks,
Meisam






--
Marcelo

Re: user defined sessionId / URI for Livy sessions

2017-09-12 Thread Bikas Saha 
Hi,

Would be good to understand how this feature would be useful other than 
providing users a handle that they can rely on as a 3rd party.

E.g. something like this may be useful in an active-active livy configuration 
because its not clear how sequential numeric id's would work in that context. 
Perhaps UUID's would also suffice for the active-active setup. Maybe the use 
case for user defined sessions could be to integrate with other software that 
could reliably use the user defined name for its internal operations instead of 
using a livy session id thats not guaranteed to be unique.

Curious as to what motivated choosing sequential numbers for livy sessions?

Bikas


From: Alex Bozarth <ajboz...@us.ibm.com>
Sent: Tuesday, September 12, 2017 2:44 PM
To: user@livy.incubator.apache.org
Cc: d...@livy.incubator.apache.org
Subject: Re: user defined sessionId / URI for Livy sessions


That's fair, I'll still withhold judgement until I see an impl though.



Alex Bozarth
Software Engineer
Spark Technology Center


E-mail: ajboz...@us.ibm.com<mailto:ajboz...@us.ibm.com>
GitHub: github.com/ajbozarth<https://github.com/ajbozarth>


505 Howard Street
San Francisco, CA 94105
United States




[Inactive hide details for Marcelo Vanzin ---09/12/2017 02:40:33 PM---BTW take 
my comment there with a grain of salt; at the tim]Marcelo Vanzin ---09/12/2017 
02:40:33 PM---BTW take my comment there with a grain of salt; at the time Livy 
was being targeted at mostly being

From: Marcelo Vanzin <van...@cloudera.com>
To: d...@livy.incubator.apache.org
Cc: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>
Date: 09/12/2017 02:40 PM
Subject: Re: user defined sessionId / URI for Livy sessions





BTW take my comment there with a grain of salt; at the time Livy was being 
targeted at mostly being hidden from users, making such a feature not make much 
sense in Livy itself. But things may have changed since then, especially as 
people started using it more.

On Mon, Sep 11, 2017 at 1:51 PM, Alex Bozarth 
<ajboz...@us.ibm.com<mailto:ajboz...@us.ibm.com>> wrote:

I would agree with Marcelo's comment the JIRA that this isn't a good feature 
for livy, but I'll take a look at your impl if you open a PR and see if it 
changes my mind.


Alex Bozarth
Software Engineer
Spark Technology Center


E-mail: ajboz...@us.ibm.com<mailto:ajboz...@us.ibm.com>
GitHub: 
github.com/ajbozarth<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ajbozarth=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=VdRly7JyucmN1BnLWxZGFJY-iemUUfqUWOlQqaGAA54=047UIBZ1bxlXg56NfOnZ2UsJMzTJTSJ34UGFbLo3GbA=>


505 Howard Street
San Francisco, CA 94105
United States




Meisam Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in adding 
this feature to Livy? I can send a PR

From: Meisam Fathi <meisam.fa...@gmail.com<mailto:meisam.fa...@gmail.com>>
To: "user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>" 
<user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>>, 
"d...@livy.incubator.apache.org<mailto:d...@livy.incubator.apache.org>" 
<d...@livy.incubator.apache.org<mailto:d...@livy.incubator.apache.org>>
Date: 09/11/2017 10:23 AM
Subject: Re: user defined sessionId / URI for Livy sessions




+ dev
Is there any interest in adding this feature to Livy? I can send a PR

Ideally, it would be helpful if we could mint a session ID with a PUT
> request, something like PUT /sessions/foobar, where "foobar" is the newly
> created sessionId.
>
> I suggest we make session names unique and nonnumeric values (to guarantee
a session name does not clash with another session name or session ID).

Design doc:
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=
JIRA ticket: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=


Thanks,
Meisam






--
Marcelo

Re: user defined sessionId / URI for Livy sessions

2017-09-12 Thread Alex Bozarth 

That's fair, I'll still withhold judgement until I see an impl though.


   
 Alex Bozarth   
   
 Software Engineer  
   
 Spark Technology Center
   

   

 

 

 
 E-mail: ajboz...@us.ibm.com
 
 GitHub: github.com/ajbozarth   
 
   505 Howard 
Street 
 San Francisco, CA 
94105 
   United 
States 

 








From:   Marcelo Vanzin <van...@cloudera.com>
To: d...@livy.incubator.apache.org
Cc: "user@livy.incubator.apache.org"
<user@livy.incubator.apache.org>
Date:   09/12/2017 02:40 PM
Subject:    Re: user defined sessionId / URI for Livy sessions



BTW take my comment there with a grain of salt; at the time Livy was being
targeted at mostly being hidden from users, making such a feature not make
much sense in Livy itself. But things may have changed since then,
especially as people started using it more.

On Mon, Sep 11, 2017 at 1:51 PM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
  I would agree with Marcelo's comment the JIRA that this isn't a good
  feature for livy, but I'll take a look at your impl if you open a PR and
  see if it changes my mind.





   
 Alex Bozarth   
   
 Software Engineer  
   
 Spark Technology Center
   

   








 E-mail: ajboz...@us.ibm.com

 GitHub: github.com/ajbozarth   

  505 Howard 
Street 
San Francisco, CA 
94105 
  United 
States 







  Meisam Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in
  adding this feature to Livy? I can send a PR

  From: Meisam Fathi <meisam.fa...@gmail.com>
  To: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>, "
  d...@livy.incubator.apache.org" <d...@livy.incubator.apache.org>
  Date: 09/11/2017 10:23 AM
  Subject: Re: user defined sessionId / URI for Livy sessions



  + dev
  Is there any interest in adding this feature to Livy? I can send a PR

  Ideally, it would be helpful if we could mint a session ID with a PUT
  > request, something like PUT /sessions/foobar, where "foobar" is the
  newly
  > created sessionId.
  >
  > I suggest we make session names unique and nonnumeric values (to
  guarantee
  a session name does not clash with another session name or session ID).

  Design doc:
  
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=

  JIRA ticket:
  
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=



  Thanks,
  Meisam








--
Marcelo

Re: user defined sessionId / URI for Livy sessions

2017-09-11 Thread Meisam Fathi 
> If we're using session name, how do we guarantee the uniqueness of this
> name?
>

If the requested session name already exist, Livy returns an error and does
not create the session.

Thanks,
Meisam

Re: user defined sessionId / URI for Livy sessions

2017-09-11 Thread Saisai Shao 
If we're using session name, how do we guarantee the uniqueness of this
name?

Thanks
Jerry

On Tue, Sep 12, 2017 at 4:51 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> I would agree with Marcelo's comment the JIRA that this isn't a good
> feature for livy, but I'll take a look at your impl if you open a PR and
> see if it changes my mind.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
> San Francisco, CA 94105
> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
> United States
> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>
>
>
> [image: Inactive hide details for Meisam Fathi ---09/11/2017 10:23:49
> AM---+ dev Is there any interest in adding this feature to Livy?]Meisam
> Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in adding
> this feature to Livy? I can send a PR
>
> From: Meisam Fathi <meisam.fa...@gmail.com>
> To: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>, "
> d...@livy.incubator.apache.org" <d...@livy.incubator.apache.org>
> Date: 09/11/2017 10:23 AM
> Subject: Re: user defined sessionId / URI for Livy sessions
> --
>
>
>
> + dev
> Is there any interest in adding this feature to Livy? I can send a PR
>
> Ideally, it would be helpful if we could mint a session ID with a PUT
> > request, something like PUT /sessions/foobar, where "foobar" is the newly
> > created sessionId.
> >
> > I suggest we make session names unique and nonnumeric values (to
> guarantee
> a session name does not clash with another session name or session ID).
>
> Design doc:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-
> 2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=
> DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-
> cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=
> xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=
> JIRA ticket: https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-
> juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=
>
>
> Thanks,
> Meisam
>
>
>
>

Re: user defined sessionId / URI for Livy sessions

2017-09-11 Thread Meisam Fathi 
+ dev
Is there any interest in adding this feature to Livy? I can send a PR

Ideally, it would be helpful if we could mint a session ID with a PUT
> request, something like PUT /sessions/foobar, where "foobar" is the newly
> created sessionId.
>
> I suggest we make session names unique and nonnumeric values (to guarantee
a session name does not clash with another session name or session ID).

Design doc:
https://github.com/meisam/incubator-livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name
JIRA ticket: https://issues.apache.org/jira/browse/LIVY-41


Thanks,
Meisam

Re: user defined sessionId / URI for Livy sessions

2017-09-06 Thread Graham Hukill 
That is ideal behavior for our purposes, thanks so much all.

-Graham

On Wed, Sep 6, 2017 at 8:52 AM, Saisai Shao  wrote:

> Session recovery should be enabled by configuration. If it is enabled,
> session id will be recorded in the persistent storage, and the number will
> continue to increment after restart.
>
> On Wed, Sep 6, 2017 at 8:27 PM, Graham Hukill  wrote:
>
>> Ah, also very good to know, thanks.  That might be a great option.
>>
>> I'm going to look into this today, but I'm curious, are sessions
>> recovered automatically?  or, if not, and a session is not recovered, does
>> the numbering continue to increment for sessionId's?
>>
>> Thanks,
>> Graham
>>
>> On Tue, Sep 5, 2017 at 9:19 PM, Vivek  wrote:
>>
>>> One other option is to turn on session recovery.
>>> This ensures that the session ids are unique
>>>
>>> Sent from my iPhone
>>>
>>> On 6 Sep 2017, at 12:10 AM, Graham Hukill  wrote:
>>>
>>> Prabhu,
>>>
>>> Thanks so much for the information, nice to know it's been formally
>>> requested.
>>>
>>> thanks,
>>> Graham
>>>
>>> On Tue, Sep 5, 2017 at 11:22 AM, Prabhu Kasinathan <
>>> vasurampra...@gmail.com> wrote:
>>>
 Livy currently doesn't have this feature now.

 Here, is the Jira for that: https://issues.cloudera.
 org/plugins/servlet/mobile#issue/LIVY-41. PayPal is already having
 that feature in production and Meisam had submitted pull request for this
 feature to Livy.

 https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-
 Accessing-sessions-by-name


 Thanks*!*
 Prabhu
 iPhone. iTypos. iApologize.

 On Sep 5, 2017, at 7:51 AM, Graham Hukill  wrote:

 I wondered if it was possible to set the session URI when creating a
 new one?  I see from the Livy REST API documentation (
 https://livy.incubator.apache.org/docs/latest/rest-api.html), under
 POST /sessions, some of the arguments that you can pass when initializing a
 new session, such as "kind", "name", etc.

 As I understand it now, sessionId's are automatically created when
 POSTing to /sessions, setting the ID as an incrementing digit from "0" when
 Livy starts.  If Livy is restarted, the numbering drops back to "0".

 Normally this would not present a problem, but we are planning on
 starting/stopping Livy sessions from another application, and could
 envision a scenario where:

- a session is started in Livy at /sessions/0
- that session ID is associated in our ancillary app with a
particular user or job
- Livy restarts, then when a new session is started, the pointer to
/sessions/0 is actually a different Livy/SparkSession

 Ideally, it would be helpful if we could mint a session ID with a PUT
 request, something like PUT /sessions/foobar, where "foobar" is the newly
 created sessionId.

 Is this currently possible in any way?  or are there undocumented
 configurations that can be passed with the POST request for creating new
 sessions that would define the sessionId?

 thanks,
 Graham


>>>
>>
>

Re: user defined sessionId / URI for Livy sessions

2017-09-06 Thread Graham Hukill 
Ah, also very good to know, thanks.  That might be a great option.

I'm going to look into this today, but I'm curious, are sessions recovered
automatically?  or, if not, and a session is not recovered, does the
numbering continue to increment for sessionId's?

Thanks,
Graham

On Tue, Sep 5, 2017 at 9:19 PM, Vivek  wrote:

> One other option is to turn on session recovery.
> This ensures that the session ids are unique
>
> Sent from my iPhone
>
> On 6 Sep 2017, at 12:10 AM, Graham Hukill  wrote:
>
> Prabhu,
>
> Thanks so much for the information, nice to know it's been formally
> requested.
>
> thanks,
> Graham
>
> On Tue, Sep 5, 2017 at 11:22 AM, Prabhu Kasinathan <
> vasurampra...@gmail.com> wrote:
>
>> Livy currently doesn't have this feature now.
>>
>> Here, is the Jira for that: https://issues.cloudera.
>> org/plugins/servlet/mobile#issue/LIVY-41. PayPal is already having that
>> feature in production and Meisam had submitted pull request for this
>> feature to Livy.
>>
>> https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-
>> Accessing-sessions-by-name
>>
>>
>> Thanks*!*
>> Prabhu
>> iPhone. iTypos. iApologize.
>>
>> On Sep 5, 2017, at 7:51 AM, Graham Hukill  wrote:
>>
>> I wondered if it was possible to set the session URI when creating a new
>> one?  I see from the Livy REST API documentation (
>> https://livy.incubator.apache.org/docs/latest/rest-api.html), under POST
>> /sessions, some of the arguments that you can pass when initializing a new
>> session, such as "kind", "name", etc.
>>
>> As I understand it now, sessionId's are automatically created when
>> POSTing to /sessions, setting the ID as an incrementing digit from "0" when
>> Livy starts.  If Livy is restarted, the numbering drops back to "0".
>>
>> Normally this would not present a problem, but we are planning on
>> starting/stopping Livy sessions from another application, and could
>> envision a scenario where:
>>
>>- a session is started in Livy at /sessions/0
>>- that session ID is associated in our ancillary app with a
>>particular user or job
>>- Livy restarts, then when a new session is started, the pointer to
>>/sessions/0 is actually a different Livy/SparkSession
>>
>> Ideally, it would be helpful if we could mint a session ID with a PUT
>> request, something like PUT /sessions/foobar, where "foobar" is the newly
>> created sessionId.
>>
>> Is this currently possible in any way?  or are there undocumented
>> configurations that can be passed with the POST request for creating new
>> sessions that would define the sessionId?
>>
>> thanks,
>> Graham
>>
>>
>

Re: user defined sessionId / URI for Livy sessions

2017-09-05 Thread Graham Hukill 
Prabhu,

Thanks so much for the information, nice to know it's been formally
requested.

thanks,
Graham

On Tue, Sep 5, 2017 at 11:22 AM, Prabhu Kasinathan 
wrote:

> Livy currently doesn't have this feature now.
>
> Here, is the Jira for that: https://issues.cloudera.
> org/plugins/servlet/mobile#issue/LIVY-41. PayPal is already having that
> feature in production and Meisam had submitted pull request for this
> feature to Livy.
>
> https://github.com/meisam/livy/wiki/Design-doc-for-Livy-
> 41:-Accessing-sessions-by-name
>
>
> Thanks*!*
> Prabhu
> iPhone. iTypos. iApologize.
>
> On Sep 5, 2017, at 7:51 AM, Graham Hukill  wrote:
>
> I wondered if it was possible to set the session URI when creating a new
> one?  I see from the Livy REST API documentation (https://livy.incubator.
> apache.org/docs/latest/rest-api.html), under POST /sessions, some of the
> arguments that you can pass when initializing a new session, such as
> "kind", "name", etc.
>
> As I understand it now, sessionId's are automatically created when POSTing
> to /sessions, setting the ID as an incrementing digit from "0" when Livy
> starts.  If Livy is restarted, the numbering drops back to "0".
>
> Normally this would not present a problem, but we are planning on
> starting/stopping Livy sessions from another application, and could
> envision a scenario where:
>
>- a session is started in Livy at /sessions/0
>- that session ID is associated in our ancillary app with a particular
>user or job
>- Livy restarts, then when a new session is started, the pointer to
>/sessions/0 is actually a different Livy/SparkSession
>
> Ideally, it would be helpful if we could mint a session ID with a PUT
> request, something like PUT /sessions/foobar, where "foobar" is the newly
> created sessionId.
>
> Is this currently possible in any way?  or are there undocumented
> configurations that can be passed with the POST request for creating new
> sessions that would define the sessionId?
>
> thanks,
> Graham
>
>