Why don't you simply use sparklyr for a more R native integration of spark? Simone Pallotta <simone.pallo...@mail.polimi.it> schrieb am So. 27. Aug. 2017 um 09:47:
> In my R code, I am using rscala package to bridge to a scala method. in > scala method I have initialized a spark context to be used later. > > > R code: > > s <- scala(classpath = "<path to jar>", heap.maximum = "4g") > > assign("WrappeR",s$.it.wrapper.r.Wrapper) > > WrappeR$init() > > where init is a scala function and Wrapper is scala class name (see below). > > I created a scala object (Wrapper) with init function > > > scala code: > > package it.wrapper.r > > object Wrapper { > > var Spark_context: SparkContext = _ > > def init(): Unit = { > > val spark_conf = new SparkConf() > > .setMaster("local[*]") > > .setAppName("GMQL-R") > > .set("spark.serializer", > "org.apache.spark.serializer.KryoSerializer") > > .set("spark.executor.memory", "6g") > > .set("spark.driver.memory", "2g") > > Spark_context = SparkContext.getOrCreate(spark_conf) > > } > > } > > I removed the following code after Spark_context because is not useful. > > this code works without flaws under OS X and Linux. Of course window is > not gentle as other OSes. I checked out my project under windows (I have > installed scala 2.11.8, java 1.8 and create environment variable JAVA_HOME > and SCALA_HOME that are mandatory for running rscala package in R, I did > the same in OS X/Linux) > > when I run the jar file as standalone application it works fine under > windows, but when I invoke the same function from R it fails giving this > error: > > Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > org.apache.hadoop.security.GroupMappingServiceProvider > > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752) > > at org.apache.hadoop.security.Groups.<init>(Groups.java:55) > > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182) > > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:235) > > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214) > > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669) > > at `enter code > here`org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571) > > at > org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) > > at > org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) > > at scala.Option.getOrElse(Option.scala:121) > > at > org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391) > > at org.apache.spark.SparkContext.<init>(SparkContext.scala:295) > > at org.apache.spark.SparkContext.<init>(SparkContext.scala:145) > > at it.polimi.genomics.r.Wrapper$.initGMQL(Wrapper.scala:98) > > at $line4.$read$$iw$$iw$$anonfun$1.apply$mcV$sp(<console>:19) > > at $line4.$read$$iw$$iw$$anonfun$1.apply(<console>:16) > > at $line4.$read$$iw$$iw$$anonfun$1.apply(<console>:16) > > ... 46 more > > Caused by: java.lang.RuntimeException: class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > org.apache.hadoop.security.GroupMappingServiceProvider > > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1746) > > ... 62 more > > java.lang.reflect.InvocationTargetException > > java.lang.RuntimeException: java.lang.RuntimeException: class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > org.apache.hadoop.security.GroupMappingServiceProvider > > > Does anyone has already seen this kind of behavior? >