[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362048#comment-17362048 ] Apache Spark commented on SPARK-35321: -- User 'sunchao' has created a pull request for this issue: https://github.com/apache/spark/pull/32887 > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362047#comment-17362047 ] Apache Spark commented on SPARK-35321: -- User 'sunchao' has created a pull request for this issue: https://github.com/apache/spark/pull/32887 > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17341396#comment-17341396 ] Dongjoon Hyun commented on SPARK-35321: --- This is reverted due to https://github.com/apache/spark/pull/32446#issuecomment-835419237 . > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17341053#comment-17341053 ] Apache Spark commented on SPARK-35321: -- User 'sunchao' has created a pull request for this issue: https://github.com/apache/spark/pull/32471 > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Fix For: 3.2.0 > > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17341056#comment-17341056 ] Apache Spark commented on SPARK-35321: -- User 'sunchao' has created a pull request for this issue: https://github.com/apache/spark/pull/32472 > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Fix For: 3.2.0 > > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339948#comment-17339948 ] Apache Spark commented on SPARK-35321: -- User 'sunchao' has created a pull request for this issue: https://github.com/apache/spark/pull/32446 > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339947#comment-17339947 ] Apache Spark commented on SPARK-35321: -- User 'sunchao' has created a pull request for this issue: https://github.com/apache/spark/pull/32446 > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339946#comment-17339946 ] Chao Sun commented on SPARK-35321: -- [~yumwang] I'm thinking of using [Hive#getWithFastCheck|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L389] for this purpose, which allows us to set the flag to false. The fast check flag also offers a way to compare {{HiveConf}} faster when the conf rarely changes. > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339944#comment-17339944 ] Yuming Wang commented on SPARK-35321: - Could we add a parameter to disable registerAllFunctionsOnce? https://issues.apache.org/jira/browse/HIVE-21563 > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339911#comment-17339911 ] Erik Krogen commented on SPARK-35321: - Gotcha. Yeah, agreed that if it's unnecessary we may as well improve the cross-version compatibility. > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339906#comment-17339906 ] Chao Sun commented on SPARK-35321: -- [~xkrogen] yes that can help to solve the issue, but users need to specify both {{spark.sql.hive.metastore.version}} and {{spark.sql.hive.metastore.jars}}. The latter is not so easy to setup: the {{maven}} option usually takes a very long time to download all the jars, while the {{path}} option require users to download all the relevant Hive jars with the specific version and it's tedious. I think this specific issue is worth fixing in Spark itself regardless since it doesn't really need to load all the permanent functions when starting up Hive client from what I can see. The process could also be pretty expensive if there are many UDFs registered in HMS. > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339904#comment-17339904 ] Erik Krogen commented on SPARK-35321: - Isn't this what the {{IsolatedClientLoader}} is for? You should use e.g. {{spark.sql.hive.metastore.version=1.1}} and this issue will go away IIUC. > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift > API missing > --- > > Key: SPARK-35321 > URL: https://issues.apache.org/jira/browse/SPARK-35321 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.2, 3.1.1, 3.2.0 >Reporter: Chao Sun >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API > {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. > This is called when creating a new {{Hive}} object: > {code} > private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException { > conf = c; > if (doRegisterAllFns) { > registerAllFunctionsOnce(); > } > } > {code} > {{registerAllFunctionsOnce}} will reload all the permanent functions by > calling the {{get_all_functions}} API from the megastore. In Spark, we always > pass {{doRegisterAllFns}} as true, and this will cause failure: > {code} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) > at > org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) > at > org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) > ... 96 more > Caused by: org.apache.thrift.TApplicationException: Invalid method name: > 'get_all_functions' > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) > {code} > It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} > since it loads the Hive permanent function directly from HMS API. The Hive > {{FunctionRegistry}} is only used for loading Hive built-in functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org