[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924894#comment-16924894 ] Marco Gaido commented on LIVY-489: -- The right place is the user mailing list, please see https://livy.apache.org/community/. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Assignee: Marco Gaido >Priority: Major > Fix For: 0.6.0 > > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924817#comment-16924817 ] Marco Gaido commented on LIVY-489: -- Yes, sure. The point is: that template in not exhaustive and does not contain all the configs which can be set. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Assignee: Marco Gaido >Priority: Major > Fix For: 0.6.0 > > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[
https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924812#comment-16924812
]
Marco Gaido commented on LIVY-489:
--
[~vonatzki] nothing to be sorry about. Please see
https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/LivyConf.scala#L101.
{{livy.server.thrift.enabled}} must be set to true, the port is controlled by
https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/LivyConf.scala#L109.
Then you should be able to connect using the beeline you can find in the repo
or with any Hive 3 client. Thanks.
> Expose a JDBC endpoint for Livy
> ---
>
> Key: LIVY-489
> URL: https://issues.apache.org/jira/browse/LIVY-489
> Project: Livy
> Issue Type: New Feature
> Components: API, Server
>Affects Versions: 0.6.0
>Reporter: Marco Gaido
>Assignee: Marco Gaido
>Priority: Major
> Fix For: 0.6.0
>
>
> Many users and BI tools use JDBC connections in order to retrieve data. As
> Livy exposes only a REST API, this is a limitation in its adoption. Hence,
> adding a JDBC endpoint may be a very useful feature, which could also make
> Livy a more attractive solution for end user to adopt.
> Moreover, currently, Spark exposes a JDBC interface, but this has many
> limitations, including that all the queries are submitted to the same
> application, therefore there is no isolation/security, which can be offered
> by Livy, making a Livy JDBC API a better solution for companies/users who
> want to use Spark in order to run they queries through JDBC.
> In order to make the transition from existing solutions to the new JDBC
> server seamless, the proposal is to use the Hive thrift-server and extend it
> as it was done by the STS.
> [Here, you can find the design
> doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit]
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922946#comment-16922946 ] Von Han Yu commented on LIVY-489: - Awesome, thanks [~mgaido]. Will check the livy.conf. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Assignee: Marco Gaido >Priority: Major > Fix For: 0.6.0 > > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922363#comment-16922363 ] Marco Gaido commented on LIVY-489: -- Hi [~vonatzki]. Please check the design doc attached here. Moreover, please check the configurations added in LivyConf. You'll see that the thriftserver must be enabled and configured properly and default values for the configurations are there. As a suggestion, it is possible for you, I'd recommend to build Livy from master branch, even though the thriftserver is present also in the 0.6.0 version. Indeed, it was the first release for it, so might find issues which have been fixed on current master. Thanks. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Assignee: Marco Gaido >Priority: Major > Fix For: 0.6.0 > > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922294#comment-16922294 ] Von Han Yu commented on LIVY-489: - Hi [~mgaido], I am interested on testing this feature but I do not know where to start. Does this assume that I can connect via JDBC in the same port of the UI? I tried using beeline but it seems it's not working. Would appreciate some guidance on this. Thanks! > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Assignee: Marco Gaido >Priority: Major > Fix For: 0.6.0 > > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581075#comment-16581075 ] Marco Gaido commented on LIVY-489: -- Sure [~jerryshao], thank you. I am submitting the first PR for 2, 3, 4. Thanks. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581056#comment-16581056 ] Saisai Shao commented on LIVY-489: -- Hi [~mgaido], let's combine task 2, 3, 4 together, and 5, 6 separately. Also please submit a PR accordingly. I've already reviewed partly, can you please submit a PR against Livy master? > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578237#comment-16578237 ] Marco Gaido commented on LIVY-489: -- [~jerryshao] I created 5 subtasks for this. Hope they are reasonable to you. Thanks. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578226#comment-16578226 ] Marco Gaido commented on LIVY-489: -- Sure [~jerryshao], the branch is https://github.com/mgaido91/incubator-livy/tree/livy_thrift, and the diff is https://github.com/apache/incubator-livy/compare/master...mgaido91:livy_thrift. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578207#comment-16578207 ] Saisai Shao commented on LIVY-489: -- BTW, [~mgaido] do you have a branch or a diff so that I can take a review. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578133#comment-16578133 ] Saisai Shao commented on LIVY-489: -- I see. But it is hard to review for such a big thing, it would be better to split to small pieces. You don't have to make them isolated, a series of dependent tasks is OK. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577995#comment-16577995 ] Marco Gaido commented on LIVY-489: -- Hi [~jerryshao]. Thanks for your comment. Unfortunately I am not sure how to spit the implementation as most of the code is required for it to work. s of now, the only two tasks I have been able to split it into are: - Initial Thriftserver implementation; - Adding a Thriftserver UI. I will keep on thinking on this anyway. Any suggestion is welcomed. Thanks. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-489) Expose a JDBC endpoint for Livy
[ https://issues.apache.org/jira/browse/LIVY-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1650#comment-1650 ] Saisai Shao commented on LIVY-489: -- Hi [~mgaido] can you please also list out the tasks you're planning to do and create subtasks accordingly. > Expose a JDBC endpoint for Livy > --- > > Key: LIVY-489 > URL: https://issues.apache.org/jira/browse/LIVY-489 > Project: Livy > Issue Type: New Feature > Components: API, Server >Affects Versions: 0.6.0 >Reporter: Marco Gaido >Priority: Major > > Many users and BI tools use JDBC connections in order to retrieve data. As > Livy exposes only a REST API, this is a limitation in its adoption. Hence, > adding a JDBC endpoint may be a very useful feature, which could also make > Livy a more attractive solution for end user to adopt. > Moreover, currently, Spark exposes a JDBC interface, but this has many > limitations, including that all the queries are submitted to the same > application, therefore there is no isolation/security, which can be offered > by Livy, making a Livy JDBC API a better solution for companies/users who > want to use Spark in order to run they queries through JDBC. > In order to make the transition from existing solutions to the new JDBC > server seamless, the proposal is to use the Hive thrift-server and extend it > as it was done by the STS. > [Here, you can find the design > doc.|https://docs.google.com/document/d/18HAR_VnQLegbYyzGg8f4zwD4GtDP5q_t3K21eXecZC4/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
