Re: Accessing Detailed Livy Session Information (session name?)
I think in general our website documentation is not complete and I think it would be great if we could improve it significantly. For instance, there is no place where all configurations are described properly (other than reading the code). I think that PRs and help in improving the doc would be highly appreciated. Thanks, Marco On Tue, 16 Apr 2019, 05:13 Meisam Fathi, wrote: > Hi Peter, > > Are you using ZooKeeper for recovery store? > If yes, in conf/livy.conf, is > livy.server.recovery.zk-state-store.key-prefix > set to different values in different Livy instances? If not, all of Livy > instances will read/write the recovery data from/to the same path, which is > default is /livy/v1 by default. > > @dev mailing list: > This behavior is not documented in livy.conf nor on the website. It might > be a good idea to document it somewhere. > > Thanks, > Meisam > > On Fri, Apr 12, 2019 at 3:20 PM Meisam Fathi > wrote: > > > Hi Peter, > > > > Livy 0.6 has a new feature to give each session a name: > > https://github.com/apache/incubator-livy/pull/48 > > > > Would this feature be useful in your usecase? > > > > Thanks, > > Meisam > > > > On Fri, Apr 12, 2019, 8:51 AM Peter Wicks (pwicks) > > wrote: > > > >> Greetings, > >> > >> > >> > >> I have a custom service that connects to Livy, v0.4 soon to be v0.5 once > >> we go to HDP3. If sessions already exist it logs the session ID’s and > >> starts using them, if sessions don’t exist it creates new ones. The > problem > >> is the account used to launch the Livy sessions is not unique to this > >> service, nor is the kind of session. So sometimes it grabs other > people’s > >> sessions and absconds off with them. Also, there are multiple instances > of > >> the service, running under the same account, and they are not supposed > to > >> use each other’s sessions… that’s not working out so well. > >> > >> > >> > >> The service names the sessions, but I can’t find any way to retrieve > >> detailed session data so that I can update the service to check if the > Livy > >> Session belongs to the service or not. > >> > >> > >> > >> I found some older comments 2016/2017 about retrieving Livy sessions by > >> name. I don’t really need that, I just want to be able to read the name > >> through the regular sessions REST call. > >> > >> > >> > >> Any REST calls I missed, or undocumented calls… that can help? > >> > >> > >> > >> Thanks, > >> > >> Peter > >> > >> > >> > >> Ref: > >> > https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name > , > >> https://issues.cloudera.org/browse/LIVY-41 > >> > >> > >> > >> > >> > > >
Re: Accessing Detailed Livy Session Information (session name?)
Hi Peter, Are you using ZooKeeper for recovery store? If yes, in conf/livy.conf, is livy.server.recovery.zk-state-store.key-prefix set to different values in different Livy instances? If not, all of Livy instances will read/write the recovery data from/to the same path, which is default is /livy/v1 by default. @dev mailing list: This behavior is not documented in livy.conf nor on the website. It might be a good idea to document it somewhere. Thanks, Meisam On Fri, Apr 12, 2019 at 3:20 PM Meisam Fathi wrote: > Hi Peter, > > Livy 0.6 has a new feature to give each session a name: > https://github.com/apache/incubator-livy/pull/48 > > Would this feature be useful in your usecase? > > Thanks, > Meisam > > On Fri, Apr 12, 2019, 8:51 AM Peter Wicks (pwicks) > wrote: > >> Greetings, >> >> >> >> I have a custom service that connects to Livy, v0.4 soon to be v0.5 once >> we go to HDP3. If sessions already exist it logs the session ID’s and >> starts using them, if sessions don’t exist it creates new ones. The problem >> is the account used to launch the Livy sessions is not unique to this >> service, nor is the kind of session. So sometimes it grabs other people’s >> sessions and absconds off with them. Also, there are multiple instances of >> the service, running under the same account, and they are not supposed to >> use each other’s sessions… that’s not working out so well. >> >> >> >> The service names the sessions, but I can’t find any way to retrieve >> detailed session data so that I can update the service to check if the Livy >> Session belongs to the service or not. >> >> >> >> I found some older comments 2016/2017 about retrieving Livy sessions by >> name. I don’t really need that, I just want to be able to read the name >> through the regular sessions REST call. >> >> >> >> Any REST calls I missed, or undocumented calls… that can help? >> >> >> >> Thanks, >> >> Peter >> >> >> >> Ref: >> https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name, >> https://issues.cloudera.org/browse/LIVY-41 >> >> >> >> >> >
Re: Accessing Detailed Livy Session Information (session name?)
Hi Peter, Livy 0.6 has a new feature to give each session a name: https://github.com/apache/incubator-livy/pull/48 Would this feature be useful in your usecase? Thanks, Meisam On Fri, Apr 12, 2019, 8:51 AM Peter Wicks (pwicks) wrote: > Greetings, > > > > I have a custom service that connects to Livy, v0.4 soon to be v0.5 once > we go to HDP3. If sessions already exist it logs the session ID’s and > starts using them, if sessions don’t exist it creates new ones. The problem > is the account used to launch the Livy sessions is not unique to this > service, nor is the kind of session. So sometimes it grabs other people’s > sessions and absconds off with them. Also, there are multiple instances of > the service, running under the same account, and they are not supposed to > use each other’s sessions… that’s not working out so well. > > > > The service names the sessions, but I can’t find any way to retrieve > detailed session data so that I can update the service to check if the Livy > Session belongs to the service or not. > > > > I found some older comments 2016/2017 about retrieving Livy sessions by > name. I don’t really need that, I just want to be able to read the name > through the regular sessions REST call. > > > > Any REST calls I missed, or undocumented calls… that can help? > > > > Thanks, > > Peter > > > > Ref: > https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name, > https://issues.cloudera.org/browse/LIVY-41 > > > > >
Accessing Detailed Livy Session Information (session name?)
Greetings, I have a custom service that connects to Livy, v0.4 soon to be v0.5 once we go to HDP3. If sessions already exist it logs the session ID's and starts using them, if sessions don't exist it creates new ones. The problem is the account used to launch the Livy sessions is not unique to this service, nor is the kind of session. So sometimes it grabs other people's sessions and absconds off with them. Also, there are multiple instances of the service, running under the same account, and they are not supposed to use each other's sessions... that's not working out so well. The service names the sessions, but I can't find any way to retrieve detailed session data so that I can update the service to check if the Livy Session belongs to the service or not. I found some older comments 2016/2017 about retrieving Livy sessions by name. I don't really need that, I just want to be able to read the name through the regular sessions REST call. Any REST calls I missed, or undocumented calls... that can help? Thanks, Peter Ref: https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name, https://issues.cloudera.org/browse/LIVY-41
