[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917811#comment-16917811 ] Oleksandr Shevchenko commented on LIVY-616: --- [~jerryshao] then How to contribute guide should be updated. [https://livy.apache.org/community/] !image-2019-08-28-17-08-21-590.png! > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery.pdf, > image-2019-08-28-17-08-21-590.png > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917809#comment-16917809 ] Oleksandr Shevchenko commented on LIVY-616: --- got it, thanks > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917755#comment-16917755 ] Saisai Shao commented on LIVY-616: -- [~oshevchenko] we don't typically assign the jira ourselves, committers will assign to whom the PR is finally merged. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907099#comment-16907099 ] Oleksandr Shevchenko commented on LIVY-616: --- [~jerryshao] I have added detailed implementation info to spec file (attached to this ticket), also added more info about hight level abstraction and API using. Added some changes with the refactoring of ZooKeeper manager to make it trait to use in the same way in ZooKeeperStateStore and LivyDiscoveryManager. Updated scaladocs. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906117#comment-16906117 ] Oleksandr Shevchenko commented on LIVY-616: --- Thank you [~jerryshao]! I will try to add more information and explanation about the solution. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906098#comment-16906098 ] Saisai Shao commented on LIVY-616: -- I will spend time on the PR itself, but I'm a little busy these days, so it may not be fast to review. Also I would suggest you to separate to small sub-tasks for the ease of review, like API design, framework abstraction, detailed implementation, and so on. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906062#comment-16906062 ] Oleksandr Shevchenko commented on LIVY-616: --- [~jerryshao] what the next step here? Do I need to change something? Could you please also review code in my PR? Currently, I'm working on related stuff and have time to make changes and the possibility for testing to finish this feature. Also, I need to decide how to use Livy, with the old configuration or use discovery feature from Apache Livy or build own version with discovering. I want to add the new feature to Apache Livy for better code/architecture quality and for reusing code/feature by community since thinking this can be useful for someone else, so want to share it and see if we can contribute to Livy if we will face some bugs and we can add some improvements/new features or we need to patch own version. Also, if this feature will be approved can propose using this in Zeppelin and create PR. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905888#comment-16905888
]
Oleksandr Shevchenko commented on LIVY-616:
---
I don't think that it can be a big blocker. Even to use LivyConf we need to
depend on livy-server jar. Only one disadvantage that we can't add it to
"livy-client-common" module to use it to create clients.
{quote}
maybe we should separate this zk discovery code path.
{quote}
Not sure that it will be useful for end-clients, we still should depend on
"livy-server" to use LivyConf in our projects. So, we can't avoid adding
"livy-server" dependency. Except that we can change code path for
LivyDiscoveryManager, but this module should depend on "livy-server" since we
need to use LivyConf and ZooKeeperManager.
Also, we need to remove LivyConf from the constructor of LivyDiscoveryManager
and create it inside just to not use "livy-server" dependency from end-user
side. But then LivyDiscoveryManager will be less flexible.
What path can you suggest to make using more convenient?
> Livy Server discovery
> -
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
> Issue Type: Improvement
> Components: Server
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: Livy Server discovery spec.pdf
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without
> setting Livy address explicitly to livy.conf. A client should set
> "livy.server.host" variable and then get it via LivyConf. The same behavior
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It
> very inconvenient when we install Livy packages on couple nodes and don't
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera
> Manager. Also, in this case, we need to have Livy configuration files on a
> node where we want to get Livy address.
> It will be very helpful if we will add Livy Server address to Zookeeper and
> expose API for clients to get Livy URL to use it in client code for REST
> calls.
> Livy already supports state saving in Zookeeper but I don't see that we store
> Livy server address somewhere. Before starting investigating and
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905734#comment-16905734 ] Saisai Shao commented on LIVY-616: -- I don't have the objections about it, just thinking about how to use it properly. Anyway in the current design, we need to depend on livy-server jar, it is not so good, maybe we should separate this zk discovery code path. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905117#comment-16905117 ] Oleksandr Shevchenko commented on LIVY-616: --- [~jerryshao] Do you have any suggestions/objections about it? > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904980#comment-16904980 ] Oleksandr Shevchenko commented on LIVY-616: --- For other languages, we can use the previous solution as before. If someone really wants to use Livy Server discovery feature from not JVM language then he can get Livy Server address directly from ZK. Just simple use "livy.zookeeper.namespace" + "livy.server.zookeeper.namespace" to read data from needed znode. Of course, it requires additional code from end-user side to read data from ZK. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904966#comment-16904966 ] Saisai Shao commented on LIVY-616: -- I think what I mean is to use http endpoint, simply using curl, but can be any other language binding. The problem is still the same (don't know the server url), then how do we handle this? What you mentioned above is to use sdk, either Java or Scala... But Livy is actually a REST endpoint, so how do we handle other scenario where we don't have sdk support? > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904955#comment-16904955
]
Oleksandr Shevchenko commented on LIVY-616:
---
[~jerryshao]
Or you mean Livy HTTP endpoint for getting Livy Server address? Without
creating LivyDiscoveryManager, with just simple 'curl'?
But to use this HTTP endpoint we already need to have Livy Server address.
So, now I see only two scenarios. Using LivyDiscoveryManager directly or use
Java/Scala client.
One more thing. Will be good if we can simplify creating of client
{code}
val address = LivyDiscoveryManager(livyConf).getServerUri()
val client = LivyClientBuilder.setURI(s"http://$address";).build
{code}
I mean don't specify uri in LivyClientBuilder in client code.
{code}
val client = LivyClientBuilder.build
{code}
We can set URI by LivyDiscoveryManager from Livy project side but it requires
adding "livy-server" dependency to "livy-client-common".
> Livy Server discovery
> -
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
> Issue Type: Improvement
> Components: Server
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: Livy Server discovery spec.pdf
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without
> setting Livy address explicitly to livy.conf. A client should set
> "livy.server.host" variable and then get it via LivyConf. The same behavior
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It
> very inconvenient when we install Livy packages on couple nodes and don't
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera
> Manager. Also, in this case, we need to have Livy configuration files on a
> node where we want to get Livy address.
> It will be very helpful if we will add Livy Server address to Zookeeper and
> expose API for clients to get Livy URL to use it in client code for REST
> calls.
> Livy already supports state saving in Zookeeper but I don't see that we store
> Livy server address somewhere. Before starting investigating and
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904937#comment-16904937
]
Oleksandr Shevchenko commented on LIVY-616:
---
{quote}
Yes, I was asking how to use HTTP request directly, do you have a related API?
{quote}
yes, I use LivyDiscoveryManager API, in this way
val discoveryManager = LivyDiscoveryManager(livyConf)
val address = discoveryManager.getServerUri()
val request = sttp
.get(uri"$address/sessions/$sessionId")
.response(asJson[LivySessionResponse])
It's client code who want to use Livy REST API directly without LivyClient.
> Livy Server discovery
> -
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
> Issue Type: Improvement
> Components: Server
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: Livy Server discovery spec.pdf
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without
> setting Livy address explicitly to livy.conf. A client should set
> "livy.server.host" variable and then get it via LivyConf. The same behavior
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It
> very inconvenient when we install Livy packages on couple nodes and don't
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera
> Manager. Also, in this case, we need to have Livy configuration files on a
> node where we want to get Livy address.
> It will be very helpful if we will add Livy Server address to Zookeeper and
> expose API for clients to get Livy URL to use it in client code for REST
> calls.
> Livy already supports state saving in Zookeeper but I don't see that we store
> Livy server address somewhere. Before starting investigating and
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904922#comment-16904922 ] Saisai Shao commented on LIVY-616: -- Yes, I was asking how to use HTTP request directly, do you have a related API? > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904919#comment-16904919 ] Oleksandr Shevchenko commented on LIVY-616: --- Using of Livy REST API directly is my original scenario when I faced configuration problems. Using LivyDiscoveryServer can automate the configuration process. I consider getting LivyServer address directly from LivyDiscoveryServer (I don't use any Scala/Java clients) and then just simply use it in HTTP requests to Livy. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904913#comment-16904913
]
Oleksandr Shevchenko commented on LIVY-616:
---
I suppose it can be done by getting Livy Server address directly from
LivyDiscoveryManager (the first scenario).
This is an example for a client code with use Livy REST API directly:
{code}
val discoveryManager = LivyDiscoveryManager(livyConf)
val address = discoveryManager.getServerUri()
val request = sttp
.get(uri"$address/sessions/$sessionId")
.response(asJson[LivySessionResponse])
{code}
To use it we need to add "livy-server" dependency for LivyConf and
LivyDiscoveryManager:
{code}
"org.apache.livy" % "livy-server" % livyVersion
{code}
We need to import this lib anyway for LivyConf even if we use the old solution
to get LivyConf.SERVER_HOST and LivyConf.SERVER_PORT.
> Livy Server discovery
> -
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
> Issue Type: Improvement
> Components: Server
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: Livy Server discovery spec.pdf
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without
> setting Livy address explicitly to livy.conf. A client should set
> "livy.server.host" variable and then get it via LivyConf. The same behavior
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It
> very inconvenient when we install Livy packages on couple nodes and don't
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera
> Manager. Also, in this case, we need to have Livy configuration files on a
> node where we want to get Livy address.
> It will be very helpful if we will add Livy Server address to Zookeeper and
> expose API for clients to get Livy URL to use it in client code for REST
> calls.
> Livy already supports state saving in Zookeeper but I don't see that we store
> Livy server address somewhere. Before starting investigating and
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904908#comment-16904908 ] Saisai Shao commented on LIVY-616: -- I think I was asking if user directly using REST API, not Livy JobAPI. How do they get the Livy host url, do you have a such design? > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904900#comment-16904900 ] Oleksandr Shevchenko commented on LIVY-616: --- [~jerryshao] 1. I considered two scenarios: - getting Livy Server address directly from LivyDiscoveryManager: val discoveryManager = LivyDiscoveryManager(livyConf) val address = discoveryManager.getServerUri() - and also getting Livy Server address from Java/Scala client. might be useful when we get Livy client from somewhere, so I added a method for getting the address. Not sure if it really needed. also notice that I added a new method to LivyClient interface "Future getServerUri()", we use Future hare actually only in RPC client, I left Future for HTTP Client to honor uniform access principle: val scalaClient = new LivyScalaClient(client) val futureUri = scalaClient.getServerUri I had a question about it before. 2. Livy Server discovery has a separate logic from the State Store, now these are two independent features. Both features now use ZooKeeperManager to communicate with ZK via Curator client (if we use ZK state store). The only common thing is "livy.zookeeper.url" property which is used for both features. FileSystem state store works as before. I don't see how Livy Discovery feature relates to the FileSystem state store. Correct me if missed something. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904863#comment-16904863 ] Saisai Shao commented on LIVY-616: -- Two questions about your design: 1. How do you handle different scenarios like directly using REST API, rather than using client API? 2. So for now this is binding to zookeeper, but Livy's recovery mode can also support HDFS, how do you support this scenario? > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900749#comment-16900749 ] Saisai Shao commented on LIVY-616: -- I see your point. Let me check the design and code then. Thanks! > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900706#comment-16900706
]
Oleksandr Shevchenko commented on LIVY-616:
---
Hi [~jerryshao].
Thanks for reviewing it.
{quote}but instead you have to maintain ZK address, what's difference between
maintaining either
{quote}
Correct, the reasonable question. I thought about it and tried different
configurations.
Using of ZK address more easily since a lot of other components typically use
ZK and we don't need to perform any additional configuration for our cluster
like adding env variables for Livy address and specifying an additional
property in livy.conf. It can provide a better user experience.
The aim here is simplifying configuration. If we already have ZK on our cluster
we can configure only one property to use ZK StateStore, Livy Server discovery,
Livy ThriftServer discovery, Livy HA etc, and reuse the same ZK cluster
configuration (env variables, additional conf files etc) the same way as we do
it for other components.
It's not convenient to use ZK as state store with configured ZK address and
then add an additional variable for Livy Server address.
When I started to use Livy and find that we can use ZK as state store I tried
to find the ability to get Livy Server address as well. I was surprised that
this functionality is missed and we should still to specify Livy address
manually even though we already use ZK. It will be more easily and conventional
to store Livy Server address in ZK as well, isn't it?
I thought that this functionality will be useful for Livy community.
Also, PR for this ticket includes some small refactoring to separate ZK
communication logic from StateStore logic. It might be helpful for subsequent
Livy HA implementation. I also interested in Livy Server HA functionality and
tried to make code and configurations adaptive and reusable for Livy HA.
What do you think?
> Livy Server discovery
> -
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
> Issue Type: Improvement
> Components: Server
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: Livy Server discovery spec.pdf
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without
> setting Livy address explicitly to livy.conf. A client should set
> "livy.server.host" variable and then get it via LivyConf. The same behavior
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It
> very inconvenient when we install Livy packages on couple nodes and don't
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera
> Manager. Also, in this case, we need to have Livy configuration files on a
> node where we want to get Livy address.
> It will be very helpful if we will add Livy Server address to Zookeeper and
> expose API for clients to get Livy URL to use it in client code for REST
> calls.
> Livy already supports state saving in Zookeeper but I don't see that we store
> Livy server address somewhere. Before starting investigating and
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900602#comment-16900602 ] Saisai Shao commented on LIVY-616: -- [~oshevchenko] you can either push an empty commit or reopen the PR to trigger travis test. I'm curious about the scenarios of this proposal. Seems like you don't want to maintain the Livy address in client side, instead asking ZK to get Livy server address. I can see there's some advantages with this proposal, but instead you have to maintain ZK address, what's difference between maintaining either? Typically server discovery mechanism is mainly used in HA scenario, where you can get the active master address from ZK, but current Livy doesn't support HA. So IMHO, I don't see it super useful for now. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898783#comment-16898783 ] Oleksandr Shevchenko commented on LIVY-616: --- Can I get permissions for contributing (Jira permissions etc)? > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898761#comment-16898761
]
Oleksandr Shevchenko commented on LIVY-616:
---
Refactored configuration to provide a more flexible mechanism for configuration:
{noformat}
# Livy Server discovery
# ZooKeeper quorum URLs, e.g. host1:port1,host2:port2
# livy.zookeeper.url =
# Name of base Livy zknode. Default livy
# livy.zookeeper.namespace = livy
# Name of Livy Server zknode. Uses livy.zookeeper.namespace as parent.
# By default will be /livy/server.uri
# livy.server.zookeeper.namespace = server.uri
# Number of trials to establish the connection to ZooKeeper quorum
# livy.server.zookeeper.connection.max.retries = 3
# Sleep time between connection retries to ZooKeeper quorum
# livy.server.zookeeper.connection.retry.interval.ms = 500
{noformat}
> Livy Server discovery
> -
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
> Issue Type: Improvement
> Components: Server
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without
> setting Livy address explicitly to livy.conf. A client should set
> "livy.server.host" variable and then get it via LivyConf. The same behavior
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It
> very inconvenient when we install Livy packages on couple nodes and don't
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera
> Manager. Also, in this case, we need to have Livy configuration files on a
> node where we want to get Livy address.
> It will be very helpful if we will add Livy Server address to Zookeeper and
> expose API for clients to get Livy URL to use it in client code for REST
> calls.
> Livy already supports state saving in Zookeeper but I don't see that we store
> Livy server address somewhere. Before starting investigating and
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898714#comment-16898714 ] Oleksandr Shevchenko commented on LIVY-616: --- Tested patch manually After starting LivyServer we can get Livy Server URI from ZooKeeper [!https://user-images.githubusercontent.com/31073930/62357180-38035e00-b51b-11e9-9744-e3894c1acea8.png!|https://user-images.githubusercontent.com/31073930/62357180-38035e00-b51b-11e9-9744-e3894c1acea8.png] [!https://user-images.githubusercontent.com/31073930/62357488-de4f6380-b51b-11e9-8ef0-eb0dbe549db2.png!|https://user-images.githubusercontent.com/31073930/62357488-de4f6380-b51b-11e9-8ef0-eb0dbe549db2.png] > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898183#comment-16898183
]
Oleksandr Shevchenko commented on LIVY-616:
---
I have added API for getting Livy Server address to LivyClient, HttpClient,
RSCClient and LivyScalaClient. I don't see a way to forward Livy address from
DiscoveryManager to HttpClient or LivyScalaClient (we need to add a dependency
on {{livy-server}} to {{client-http}} in this case, I'm not sure that this is
acceptable).
So, from client code we can use:
{code}
{{val address = DiscoveryManager.getServerUri() - here we can use it in REST
requests }}
{{val client = LivyClientBuilder.setURI(s"http://address";).build - or create
client }}
{{val futureUri = client.getServerUri - and get address from client if needed}}
{{val scalaClient = new LivyScalaClient(client) }}
{{val futureUri = scalaClient.getServerUri}}
{{{code}}}
Added scaladocs, small refactoring.
Could someone review my PR?
Thanks!
> Livy Server discovery
> -
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
> Issue Type: Improvement
> Components: Server
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without
> setting Livy address explicitly to livy.conf. A client should set
> "livy.server.host" variable and then get it via LivyConf. The same behavior
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It
> very inconvenient when we install Livy packages on couple nodes and don't
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera
> Manager. Also, in this case, we need to have Livy configuration files on a
> node where we want to get Livy address.
> It will be very helpful if we will add Livy Server address to Zookeeper and
> expose API for clients to get Livy URL to use it in client code for REST
> calls.
> Livy already supports state saving in Zookeeper but I don't see that we store
> Livy server address somewhere. Before starting investigating and
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16897368#comment-16897368 ] Oleksandr Shevchenko commented on LIVY-616: --- PR was opened [https://github.com/apache/incubator-livy/pull/189]. I have separated ZooKeeper logic from StateStore logic, as a result, we can use new ZooKeeperManager just to communicate with ZooKeeper and build new logic based on it (as I do for DiscoveryManager and ZooKeeperStateStore). Didn't expose new API yet for subsequent testing. What the best way to add new API? I see the couple of ways to do that: # Add it to LivyConf - in this case, we need to create DiscoveryManager in LivyConf which is not good. # Add it to LivyClient interface and set some variable during creating client via LivyClientBuilder. Please, advise. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
