I have did it as you suggested and it seems to start the jobs OK and I
see the sessions in UI but while it is being computed (I see the job
is distributed on two spark slaves where in front of that there is
spark-master), I am computing this from my localhost:
@RunWith(JUnit4.class)
public class LivyTestCase {
private static final int SAMPLES = 10000;
private static final String LIVY_URI = "http://spark-master:8998";;
@Test
public void testPiJob() throws Exception {
LivyClient client = new LivyClientBuilder()
.setURI(new URI(LIVY_URI))
.build();
final Double result = client.submit(new PiJob(1000)).get();
System.out.println(result);
}
}
It is a PiJob from sites examples.
Now what I see in Livy logs is this
org.apache.livy.shaded.kryo.kryo.KryoException: Unable to find class: PiJob
at
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
at
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656)
at org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767)
at org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63)
at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39)
at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27)
at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57)
at org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:42)
at org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: PiJob
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
... 13 more
I know to read it - I understand there is not PiJob on the class path,
but why? I have just sent that class to Livy ...
On Thu, Oct 26, 2017 at 4:17 PM, Saisai Shao <sai.sai.s...@gmail.com> wrote:
> You can choose to set "livy.spark.master" to "local" and
> "livy.spark.deploy-mode" to "client" to start Spark with local mode, in such
> case YARN is not required.
>
> Otherwise if you plan to run on YARN, you have to install Hadoop and
> configure HADOOP_CONF_DIR in livy-env.sh.
>
> On Thu, Oct 26, 2017 at 9:40 PM, Stefan Miklosovic <mikloso...@gmail.com>
> wrote:
>>
>> Hi,
>>
>> I am running Livy server in connection with Spark without Hadoop. I am
>> setting only SPARK_HOME and I am getting this in Livy UI logs after
>> job submission.
>>
>> I am using pretty much standard configuration but
>> livy.spark.deploy-mode = cluster
>>
>> Do I need to run with Hadoop installation as well and specify
>> HADOOP_CONF_DIR?
>>
>> Is not it possible to run Livy with "plain" Spark without YARN?
>>
>> stderr:
>> java.lang.ClassNotFoundException:
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:348)
>> at org.apache.spark.util.Utils$.classForName(Utils.scala:230)
>> at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:712)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> Thanks!
>>
>> --
>> Stefan Miklosovic
>
>
--
Stefan Miklosovic
