[
https://issues.apache.org/jira/browse/SPARK-29086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angerszhu updated SPARK-29086:
------------------------------
Summary: Use added jar's class as Serde class, SparkGetColumnsOperation
return empty columns (was: In jdk11, SparkGetColumnsOperation return empty
columns)
> Use added jar's class as Serde class, SparkGetColumnsOperation return empty
> columns
> -----------------------------------------------------------------------------------
>
> Key: SPARK-29086
> URL: https://issues.apache.org/jira/browse/SPARK-29086
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: angerszhu
> Priority: Major
>
> In jdk11,
> When we create a table with jar added by 'ADD JAR' sql.
> When we restart again, we use !columns table_name, we get empty seq of column:
> {code:java}
> 0: jdbc:hive2://localhost:10000/default> add jar
> /Users/angerszhu/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar;
> INFO : Added
> [/Users/angerszhu/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar]
> to class path
> INFO : Added resources:
> [/Users/angerszhu/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar]
> +---------+
> | result |
> +---------+
> +---------+
> No rows selected (0.268 seconds)
> 0: jdbc:hive2://localhost:10000/default> CREATE TABLE addJar18(key string)
> ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
> +---------+
> | Result |
> +---------+
> +---------+
> No rows selected (0.444 seconds)
> 0: jdbc:hive2://localhost:10000/default> !columns addJar18
> +------------+--------------+-------------+--------------+------------+------------+--------------+----------------+-----------------+-----------------+-----------+----------+-------------+----------------+-------------------+--------------------+-------------------+--------------+----------------+---------------+--------------+-------------------+--------------------+
> | TABLE_CAT | TABLE_SCHEM | TABLE_NAME | COLUMN_NAME | DATA_TYPE |
> TYPE_NAME | COLUMN_SIZE | BUFFER_LENGTH | DECIMAL_DIGITS | NUM_PREC_RADIX
> | NULLABLE | REMARKS | COLUMN_DEF | SQL_DATA_TYPE | SQL_DATETIME_SUB |
> CHAR_OCTET_LENGTH | ORDINAL_POSITION | IS_NULLABLE | SCOPE_CATALOG |
> SCOPE_SCHEMA | SCOPE_TABLE | SOURCE_DATA_TYPE | IS_AUTO_INCREMENT |
> +------------+--------------+-------------+--------------+------------+------------+--------------+----------------+-----------------+-----------------+-----------+----------+-------------+----------------+-------------------+--------------------+-------------------+--------------+----------------+---------------+--------------+-------------------+--------------------+
> | NULL | default | addjar18 | key | 12 |
> STRING | NULL | NULL | NULL | NULL
> | 1 | | NULL | NULL | NULL |
> NULL | NULL | YES | NULL | NULL
> | NULL | NULL | NO |
> +------------+--------------+-------------+--------------+------------+------------+--------------+----------------+-----------------+-----------------+-----------+----------+-------------+----------------+-------------------+--------------------+-------------------+--------------+----------------+---------------+--------------+-------------------+--------------------+
> 0: jdbc:hive2://localhost:10000/default> exit
> {code}
> Then we restart Spark thrift server reconnect to it:
> {code:java}
> 0: jdbc:hive2://localhost:10000/default> select * from addJar18;
> Error: Error running query: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
> (state=,code=0)
> 0: jdbc:hive2://localhost:10000/default> !columns addJar18
> +------------+--------------+-------------+--------------+------------+------------+--------------+----------------+-----------------+-----------------+-----------+----------+-------------+----------------+-------------------+--------------------+-------------------+--------------+----------------+---------------+--------------+-------------------+--------------------+
> | TABLE_CAT | TABLE_SCHEM | TABLE_NAME | COLUMN_NAME | DATA_TYPE |
> TYPE_NAME | COLUMN_SIZE | BUFFER_LENGTH | DECIMAL_DIGITS | NUM_PREC_RADIX
> | NULLABLE | REMARKS | COLUMN_DEF | SQL_DATA_TYPE | SQL_DATETIME_SUB |
> CHAR_OCTET_LENGTH | ORDINAL_POSITION | IS_NULLABLE | SCOPE_CATALOG |
> SCOPE_SCHEMA | SCOPE_TABLE | SOURCE_DATA_TYPE | IS_AUTO_INCREMENT |
> +------------+--------------+-------------+--------------+------------+------------+--------------+----------------+-----------------+-----------------+-----------+----------+-------------+----------------+-------------------+--------------------+-------------------+--------------+----------------+---------------+--------------+-------------------+--------------------+
> +------------+--------------+-------------+--------------+------------+------------+--------------+----------------+-----------------+-----------------+-----------+----------+-------------+----------------+-------------------+--------------------+-------------------+--------------+----------------+---------------+--------------+-------------------+--------------------+
> 0: jdbc:hive2://localhost:10000/default> add jar
> /Users/angerszhu/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar;
> INFO : Added
> [/Users/angerszhu/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar]
> to class path
> INFO : Added resources:
> [/Users/angerszhu/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar]
> +---------+
> | result |
> +---------+
> +---------+
> No rows selected (0.08 seconds)
> 0: jdbc:hive2://localhost:10000/default> !columns addJar18
> +------------+--------------+-------------+--------------+------------+------------+--------------+----------------+-----------------+-----------------+-----------+----------+-------------+----------------+-------------------+--------------------+-------------------+--------------+----------------+---------------+--------------+-------------------+--------------------+
> | TABLE_CAT | TABLE_SCHEM | TABLE_NAME | COLUMN_NAME | DATA_TYPE |
> TYPE_NAME | COLUMN_SIZE | BUFFER_LENGTH | DECIMAL_DIGITS | NUM_PREC_RADIX
> | NULLABLE | REMARKS | COLUMN_DEF | SQL_DATA_TYPE | SQL_DATETIME_SUB |
> CHAR_OCTET_LENGTH | ORDINAL_POSITION | IS_NULLABLE | SCOPE_CATALOG |
> SCOPE_SCHEMA | SCOPE_TABLE | SOURCE_DATA_TYPE | IS_AUTO_INCREMENT |
> +------------+--------------+-------------+--------------+------------+------------+--------------+----------------+-----------------+-----------------+-----------+----------+-------------+----------------+-------------------+--------------------+-------------------+--------------+----------------+---------------+--------------+-------------------+--------------------+
> +------------+--------------+-------------+--------------+------------+------------+--------------+----------------+-----------------+-----------------+-----------+----------+-------------+----------------+-------------------+--------------------+-------------------+--------------+----------------+---------------+--------------+-------------------+--------------------+
> BackGround Error :
> 19/09/14 15:58:12 HiveServer2-Handler-Pool: Thread-78 ERROR Table: Unable to
> get field from serde: org.apache.hive.hcatalog.data.JsonSerDe19/09/14
> 15:58:12 HiveServer2-Handler-Pool: Thread-78 ERROR Table: Unable to get field
> from serde:
> org.apache.hive.hcatalog.data.JsonSerDejava.lang.RuntimeException:
> MetaException(message:java.lang.ClassNotFoundException Class
> org.apache.hive.hcatalog.data.JsonSerDe not found) at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:283)
> at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:263)
> at org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:641)
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:624) at
> org.apache.spark.sql.hive.client.HiveClientImpl.convertHiveTableToCatalogTable(HiveClientImpl.scala:426)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$2(HiveClientImpl.scala:413)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> scala.collection.TraversableLike.map(TraversableLike.scala:237) at
> scala.collection.TraversableLike.map$(TraversableLike.scala:230) at
> scala.collection.AbstractTraversable.map(Traversable.scala:108) at
> org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$1(HiveClientImpl.scala:413)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:311)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:245)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:244)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:294)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.getTablesByName(HiveClientImpl.scala:413)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.getRawTablesByNames(HiveExternalCatalog.scala:124)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$getTablesByName$1(HiveExternalCatalog.scala:711)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:99)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.getTablesByName(HiveExternalCatalog.scala:711)
> at
> org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.getTablesByName(ExternalCatalogWithListener.scala:142)
> at
> org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTablesByName(SessionCatalog.scala:460)
> at
> org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.$anonfun$runInternal$3(SparkGetColumnsOperation.scala:110)
> at
> org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.$anonfun$runInternal$3$adapted(SparkGetColumnsOperation.scala:106)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:128) at
> org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.runInternal(SparkGetColumnsOperation.scala:106)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:275)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.getColumns(HiveSessionImpl.java:615)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.base/java.security.AccessController.doPrivileged(Native Method) at
> java.base/javax.security.auth.Subject.doAs(Subject.java:423) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy23.getColumns(Unknown Source) at
> org.apache.hive.service.cli.CLIService.getColumns(CLIService.java:386) at
> org.apache.hive.service.cli.thrift.ThriftCLIService.GetColumns(ThriftCLIService.java:526)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetColumns.getResult(TCLIService.java:1557)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetColumns.getResult(TCLIService.java:1542)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)Caused by:
> MetaException(message:java.lang.ClassNotFoundException Class
> org.apache.hive.hcatalog.data.JsonSerDe not found) at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:458)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:437)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281)
> ... 53 more19/0{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
