I have been unsuccessful with incorporating an external Jar into a SparkR program. Does anyone know how to do this successfully?
JarTest.java ================= package com.myco; public class JarTest { public static double myStaticMethod() { return 5.515; } } ================= JarTest.R ================= Sys.setenv(SPARK_HOME="/usr/local/spark-1.4.0-bin-hadoop2.6/") .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths())) library(SparkR) #sparkR.stop() # Stop if you want to rerun the code sc = sparkR.init(master="local", sparkJars=c("/locOfMyJar/JarTest.jar")) SparkR:::callJStatic("java.lang.Math", "max", 5, 2) # OK: 5 SparkR:::callJStatic("java.lang.Math", "min", 5, 2) # OK: 2 SparkR:::callJStatic("com.myco.JarTest", "myStaticMethod") # Fails, see below # 5/06/22 13:38:07 ERROR RBackendHandler: myStaticMethod on com.myco.JarTest failed # java.lang.ClassNotFoundException: com.myco.JarTest # at java.net.URLClassLoader$1.run(URLClassLoader.java:366) # at java.net.URLClassLoader$1.run(URLClassLoader.java:355) # at java.security.AccessController.doPrivileged(Native Method) # at java.net.URLClassLoader.findClass(URLClassLoader.java:354) # at java.lang.ClassLoader.loadClass(ClassLoader.java:425) # at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) # at java.lang.ClassLoader.loadClass(ClassLoader.java:358) # at java.lang.Class.forName0(Native Method) # at java.lang.Class.forName(Class.java:190) # at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:101) # at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:74) # at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:36) # at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) # at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) # at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) # at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) # at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) # at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) # at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) # at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) # at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) # at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) # at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) # at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) # at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) # at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) # at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) # at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) # at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) # at java.lang.Thread.run(Thread.java:744) # Error: returnStatus == 0 is not TRUE ================= So basically something like: > javac jarTest.java > jar -cf JarTest.jar JarTest.class > jar -tf JarTest.jar Then I run RStudio or R with the commands you see in JarTest.R (make sure you point to your jar file). As you can see in the comments, it does not appear to find the Java class. Does anyone know a way to make this work correctly? Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/External-Jar-file-with-SparkR-tp23433.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org