[
https://issues.apache.org/jira/browse/LIVY-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marco Gaido updated LIVY-572:
-----------------------------
Description:
When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is by
default, and Spark jars are not available in Livy JVM, all operations fail due
to:
{code}
19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
at
org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
at
org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
at
org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
at
org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
at
org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
at
org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
{code}
was:
When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is by
default, and Spark jars are not available in Livy JVM, all operations fail due
to:
{{code}}
19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
at
org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
at
org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
at
org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
at
org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
at
org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
at
org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
at
org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
{{code}}
> Log operations fails because of dependencies on Spark
> -----------------------------------------------------
>
> Key: LIVY-572
> URL: https://issues.apache.org/jira/browse/LIVY-572
> Project: Livy
> Issue Type: Bug
> Components: Thriftserver
> Reporter: Marco Gaido
> Priority: Major
>
> When {{livy.server.thrift.logging.operation.enabled}} is {{true}}, which is
> by default, and Spark jars are not available in Livy JVM, all operations fail
> due to:
> {code}
> 19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session:
> SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
> Exception in thread "LivyThriftserver-Handler-Pool: Thread-30"
> java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
> at
> org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
> at
> org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
> at
> org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
> at
> org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
> at
> org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
> at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
> at
> org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
> at
> org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
> at
> org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
> at
> org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 21 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
