[
https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angerszhu updated SPARK-29018:
------------------------------
Description:
With the development of Spark and Hive，in current `sql/hive-thriftserver`, we
need to do a lot of work to solve code conflicts between different hive
versions. It's an annoying and unending work in current ways. And these issues
are troubling us when we develop new features for the SparkThriftServer2. We
suppose to implement a new thrift server based on latest v11
`TCLService.thrift` thrift protocol. Implement all API in spark's own code to
get rid of hive code .
Finally, the new thrift server have below feature:
# support all functions current `hive-thriftserver` support
# use all code maintained by spark itself
# realize origin function fit to spark’s own feature, won't limited by hive's
code
# support running without hive metastore or with hive metastore
# support user impersonation by Multi-tenant authority separation, hive
authentication and DFS authentication
# add a new module `spark-jdbc`, with connection url
`jdbc:spark:<host>:<port>/<db>`, all `hive-jdbc` support we will all support
# support both `hive-jdbc` and `spark-jdbc` client for compatibility with most
clients
We have done all these works in our repo, now we plan merge our code into the
master step by step.
# *phase1* pr about build new module `spark-service` on folder `sql/service`
2. *phase2* pr thrift protocol and generated thrift protocol java code
3. *phase3* pr with all `spark-service` module code and description about
design, also UT
4. *phase4* pr about build new module `spark-jdbc` on folder `sql/jdbc`
5. *phase5* pr with all `spark-jdbc` module code and UT
6. *phase6* pr about support thriftserver Impersonation
7. *phase7* pr about build spark's own beeline client `spark-beeline`
8. *phase8* pr about spark's own Cli client `Spark SQL CLI` `spark-cli`
Google Doc :
[https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.im79wwkzycsr]
was:
With the development of Spark and Hive，in current `sql/hive-thriftserver`, we
need to do a lot of work to solve code conflicts between different hive
versions. It's an annoying and unending work in current ways. And these issues
are troubling us when we develop new features for the SparkThriftServer2. We
suppose to implement a new thrift server based on latest v11
`TCLService.thrift` thrift protocol. Implement all API in spark's own code to
get rid of hive code .
Finally, the new thrift server have below feature:
# support all functions current `hive-thriftserver` support
# use all code maintained by spark itself
# realize origin function fit to spark’s own feature, won't limited by hive's
code
# support running without hive metastore or with hive metastore
# support user impersonation by Multi-tenant authority separation, hive
authentication and DFS authentication
# add a new module `spark-jdbc`, with connection url
`jdbc:spark:<host>:<port>/<db>`, all `hive-jdbc` support we will all support
# support both `hive-jdbc` and `spark-jdbc` client for compatibility with most
clients
We have done all these works in our repo, now we plan merge our code into the
master step by step.
1. *phase1* pr about build new module `spark-service` on folder `sql/service`
2. *phase2* pr thrift protocol and generated thrift protocol java code
3. *phase3* pr with all `spark-service` module code and description about
design, also UT
4. *phase4* pr about build new module `spark-jdbc` on folder `sql/jdbc`
5. *phase5* pr with all `spark-jdbc` module code and UT
6. *phase6* pr about support thriftserver Impersonation
7. *phase7* pr about build spark's own beeline client `spark-beeline`
8. *phase8* pr about spark's own Cli client `Spark SQL CLI` `spark-cli`
> Spark ThriftServer change to it's own API
> -----------------------------------------
>
> Key: SPARK-29018
> URL: https://issues.apache.org/jira/browse/SPARK-29018
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: angerszhu
> Priority: Major
>
> With the development of Spark and Hive，in current `sql/hive-thriftserver`, we
> need to do a lot of work to solve code conflicts between different hive
> versions. It's an annoying and unending work in current ways. And these
> issues are troubling us when we develop new features for the
> SparkThriftServer2. We suppose to implement a new thrift server based on
> latest v11 `TCLService.thrift` thrift protocol. Implement all API in spark's
> own code to get rid of hive code .
> Finally, the new thrift server have below feature:
> # support all functions current `hive-thriftserver` support
> # use all code maintained by spark itself
> # realize origin function fit to spark’s own feature, won't limited by
> hive's code
> # support running without hive metastore or with hive metastore
> # support user impersonation by Multi-tenant authority separation, hive
> authentication and DFS authentication
> # add a new module `spark-jdbc`, with connection url
> `jdbc:spark:<host>:<port>/<db>`, all `hive-jdbc` support we will all support
> # support both `hive-jdbc` and `spark-jdbc` client for compatibility with
> most clients
> We have done all these works in our repo, now we plan merge our code into the
> master step by step.
> # *phase1* pr about build new module `spark-service` on folder `sql/service`
> 2. *phase2* pr thrift protocol and generated thrift protocol java code
> 3. *phase3* pr with all `spark-service` module code and description about
> design, also UT
> 4. *phase4* pr about build new module `spark-jdbc` on folder `sql/jdbc`
> 5. *phase5* pr with all `spark-jdbc` module code and UT
> 6. *phase6* pr about support thriftserver Impersonation
> 7. *phase7* pr about build spark's own beeline client `spark-beeline`
> 8. *phase8* pr about spark's own Cli client `Spark SQL CLI` `spark-cli`
>
>
> Google Doc :
> [https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.im79wwkzycsr]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
