The problem is my jar file is big(250m) and LivyScalaClient fails to upload.
The same error reported at
https://community.cloudera.com/t5/Support-Questions/SizeConstraintExceededException-on-Using-Livy-jar-file-size/td-p/242223.
Any suggestion?
Thanks,
Feng
From: XU Feng
Sent: Sunday, August 22, 2021 11:38 AM
To: user@livy.apache.org
Subject: Question about LivyScalaClient
I am trying to use LivyScalaClient 0.7.1 with HDP 3.1.1. My Spark Application
is packaged into a big self-contained jar file that does not have dependency on
any other jar files, and the jar file is put into spark.yarn.archive directory.
The job fails in Yarn with the following exception,
21/08/22 11:41:07 INFO JobWrapper: Failed to run job
4aafeaad-b331-4f55-9cf5-5976f542a99c
org.apache.livy.shaded.kryo.kryo.KryoException: Unable to find class:
org.apache.livy.scalaapi.LivyScalaClient$$anon$1
at
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
at
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656)
at
org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767)
at
org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63)
at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39)
at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27)
at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
at
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:45)
at
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.livy.scalaapi.LivyScalaClient$$anon$1
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
... 13 more
I am sure the Spark application jar file contains
org.apache.livy.scalaapi.LivyScalaClient.
Thanks,
Feng
