Re: Kill Spark Streaming JOB from Spark UI or Yarn
The batches should all have the same application ID, so use that one. You can also find the application in the YARN UI to terminate it from there. Matei > On Aug 27, 2017, at 10:27 AM, KhajaAsmath Mohammed> wrote: > > Hi, > > I am new to spark streaming and not able to find an option to kill it after > starting spark streaming context. > > Streaming Tab doesnt have option to kill it. > > Jobs tab too doesn't have option to kill it > > > > if scheduled on yarn, how to kill that if spark submit is running in > background as I will not have an option to find yarn application id. does > batches have separate yarn application id or same one? > > Thanks, > Asmath - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: different behaviour linux/Unix vs windows when load spark context in scala method called from R function using rscala package
Why don't you simply use sparklyr for a more R native integration of spark? Simone Pallottaschrieb am So. 27. Aug. 2017 um 09:47: > In my R code, I am using rscala package to bridge to a scala method. in > scala method I have initialized a spark context to be used later. > > > R code: > > s <- scala(classpath = "", heap.maximum = "4g") > > assign("WrappeR",s$.it.wrapper.r.Wrapper) > > WrappeR$init() > > where init is a scala function and Wrapper is scala class name (see below). > > I created a scala object (Wrapper) with init function > > > scala code: > > package it.wrapper.r > > object Wrapper { > > var Spark_context: SparkContext = _ > > def init(): Unit = { > > val spark_conf = new SparkConf() > > .setMaster("local[*]") > > .setAppName("GMQL-R") > > .set("spark.serializer", > "org.apache.spark.serializer.KryoSerializer") > > .set("spark.executor.memory", "6g") > > .set("spark.driver.memory", "2g") > > Spark_context = SparkContext.getOrCreate(spark_conf) > > } > > } > > I removed the following code after Spark_context because is not useful. > > this code works without flaws under OS X and Linux. Of course window is > not gentle as other OSes. I checked out my project under windows (I have > installed scala 2.11.8, java 1.8 and create environment variable JAVA_HOME > and SCALA_HOME that are mandatory for running rscala package in R, I did > the same in OS X/Linux) > > when I run the jar file as standalone application it works fine under > windows, but when I invoke the same function from R it fails giving this > error: > > Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > org.apache.hadoop.security.GroupMappingServiceProvider > > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752) > > at org.apache.hadoop.security.Groups.(Groups.java:55) > > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182) > > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:235) > > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214) > > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669) > > at `enter code > here`org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571) > > at > org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) > > at > org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) > > at scala.Option.getOrElse(Option.scala:121) > > at > org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391) > > at org.apache.spark.SparkContext.(SparkContext.scala:295) > > at org.apache.spark.SparkContext.(SparkContext.scala:145) > > at it.polimi.genomics.r.Wrapper$.initGMQL(Wrapper.scala:98) > > at $line4.$read$$iw$$iw$$anonfun$1.apply$mcV$sp(:19) > > at $line4.$read$$iw$$iw$$anonfun$1.apply(:16) > > at $line4.$read$$iw$$iw$$anonfun$1.apply(:16) > > ... 46 more > > Caused by: java.lang.RuntimeException: class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > org.apache.hadoop.security.GroupMappingServiceProvider > > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1746) > > ... 62 more > > java.lang.reflect.InvocationTargetException > > java.lang.RuntimeException: java.lang.RuntimeException: class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > org.apache.hadoop.security.GroupMappingServiceProvider > > > Does anyone has already seen this kind of behavior? >
different behaviour linux/Unix vs windows when load spark context in scala method called from R function using rscala package
In my R code, I am using rscala package to bridge to a scala method. in scala method I have initialized a spark context to be used later. R code: s <- scala(classpath = "", heap.maximum = "4g") assign("WrappeR",s$.it.wrapper.r.Wrapper) WrappeR$init() where init is a scala function and Wrapper is scala class name (see below). I created a scala object (Wrapper) with init function scala code: package it.wrapper.r object Wrapper { var Spark_context: SparkContext = _ def init(): Unit = { val spark_conf = new SparkConf() .setMaster("local[*]") .setAppName("GMQL-R") .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") .set("spark.executor.memory", "6g") .set("spark.driver.memory", "2g") Spark_context = SparkContext.getOrCreate(spark_conf) } } I removed the following code after Spark_context because is not useful. this code works without flaws under OS X and Linux. Of course window is not gentle as other OSes. I checked out my project under windows (I have installed scala 2.11.8, java 1.8 and create environment variable JAVA_HOME and SCALA_HOME that are mandatory for running rscala package in R, I did the same in OS X/Linux) when I run the jar file as standalone application it works fine under windows, but when I invoke the same function from R it fails giving this error: Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752) at org.apache.hadoop.security.Groups.(Groups.java:55) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:235) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669) at `enter code here`org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391) at org.apache.spark.SparkContext.(SparkContext.scala:295) at org.apache.spark.SparkContext.(SparkContext.scala:145) at it.polimi.genomics.r.Wrapper$.initGMQL(Wrapper.scala:98) at $line4.$read$$iw$$iw$$anonfun$1.apply$mcV$sp(:19) at $line4.$read$$iw$$iw$$anonfun$1.apply(:16) at $line4.$read$$iw$$iw$$anonfun$1.apply(:16) ... 46 more Caused by: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1746) ... 62 more java.lang.reflect.InvocationTargetException java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider Does anyone has already seen this kind of behavior?