It would be nice to see the code for MapR FS Java API, but my google foo failed me (assuming it's open source)...
So, shooting in the dark ;) there are a few things I would check, if you haven't already: 1. Could there be 1.2 versions of some Spark jars that get picked up at run time (but apparently not in local mode) on one or more nodes? (Side question: Does your node experiment fail on all nodes?) Put another way, are the classpaths good for all JVM tasks? 2. Can you use just MapR and Spark 1.3.1 successfully, bypassing Mesos? Incidentally, how are you combining Mesos and MapR? Are you running Spark in Mesos, but accessing data in MapR-FS? Perhaps the MapR "shim" library doesn't support Spark 1.3.1. HTH, dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Mon, Jun 1, 2015 at 2:49 PM, John Omernik <j...@omernik.com> wrote: > All - > > I am facing and odd issue and I am not really sure where to go for support > at this point. I am running MapR which complicates things as it relates to > Mesos, however this HAS worked in the past with no issues so I am stumped > here. > > So for starters, here is what I am trying to run. This is a simple show > tables using the Hive Context: > > from pyspark import SparkContext, SparkConf > from pyspark.sql import SQLContext, Row, HiveContext > sparkhc = HiveContext(sc) > test = sparkhc.sql("show tables") > for r in test.collect(): > print r > > When I run it on 1.3.1 using ./bin/pyspark --master local This works with > no issues. > > When I run it using Mesos with all the settings configured (as they had > worked in the past) I get lost tasks and when I zoom in them, the error > that is being reported is below. Basically it's a NullPointerException on > the com.mapr.fs.ShimLoader. What's weird to me is is I took each instance > and compared both together, the class path, everything is exactly the same. > Yet running in local mode works, and running in mesos fails. Also of note, > when the task is scheduled to run on the same node as when I run locally, > that fails too! (Baffling). > > Ok, for comparison, how I configured Mesos was to download the mapr4 > package from spark.apache.org. Using the exact same configuration file > (except for changing the executor tgz from 1.2.0 to 1.3.1) from the 1.2.0. > When I run this example with the mapr4 for 1.2.0 there is no issue in > Mesos, everything runs as intended. Using the same package for 1.3.1 then > it fails. > > (Also of note, 1.2.1 gives a 404 error, 1.2.2 fails, and 1.3.0 fails as > well). > > So basically When I used 1.2.0 and followed a set of steps, it worked on > Mesos and 1.3.1 fails. Since this is a "current" version of Spark, MapR is > supports 1.2.1 only. (Still working on that). > > I guess I am at a loss right now on why this would be happening, any > pointers on where I could look or what I could tweak would be greatly > appreciated. Additionally, if there is something I could specifically draw > to the attention of MapR on this problem please let me know, I am perplexed > on the change from 1.2.0 to 1.3.1. > > Thank you, > > John > > > > > Full Error on 1.3.1 on Mesos: > 15/05/19 09:31:26 INFO MemoryStore: MemoryStore started with capacity > 1060.3 MB java.lang.NullPointerException at > com.mapr.fs.ShimLoader.getRootClassLoader(ShimLoader.java:96) at > com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:232) at > com.mapr.fs.ShimLoader.load(ShimLoader.java:194) at > org.apache.hadoop.conf.CoreDefaultProperties.(CoreDefaultProperties.java:60) > at java.lang.Class.forName0(Native Method) at > java.lang.Class.forName(Class.java:274) at > org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1847) > at > org.apache.hadoop.conf.Configuration.getProperties(Configuration.java:2062) > at > org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2272) > at > org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2224) > at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2141) > at org.apache.hadoop.conf.Configuration.set(Configuration.java:992) at > org.apache.hadoop.conf.Configuration.set(Configuration.java:966) at > org.apache.spark.deploy.SparkHadoopUtil.newConfiguration(SparkHadoopUtil.scala:98) > at org.apache.spark.deploy.SparkHadoopUtil.(SparkHadoopUtil.scala:43) at > org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala:220) at > org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala) at > org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1959) at > org.apache.spark.storage.BlockManager.(BlockManager.scala:104) at > org.apache.spark.storage.BlockManager.(BlockManager.scala:179) at > org.apache.spark.SparkEnv$.create(SparkEnv.scala:310) at > org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:186) at > org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:70) > java.lang.RuntimeException: Failure loading MapRClient. at > com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:283) at > com.mapr.fs.ShimLoader.load(ShimLoader.java:194) at > org.apache.hadoop.conf.CoreDefaultProperties.(CoreDefaultProperties.java:60) > at java.lang.Class.forName0(Native Method) at > java.lang.Class.forName(Class.java:274) at > org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1847) > at > org.apache.hadoop.conf.Configuration.getProperties(Configuration.java:2062) > at > org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2272) > at > org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2224) > at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2141) > at org.apache.hadoop.conf.Configuration.set(Configuration.java:992) at > org.apache.hadoop.conf.Configuration.set(Configuration.java:966) at > org.apache.spark.deploy.SparkHadoopUtil.newConfiguration(SparkHadoopUtil.scala:98) > at org.apache.spark.deploy.SparkHadoopUtil.(SparkHadoopUtil.scala:43) at > org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala:220) at > org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala) at > org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1959) at > org.apache.spark.storage.BlockManager.(BlockManager.scala:104) at > org.apache.spark.storage.BlockManager.(BlockManager.scala:179) at > org.apache.spark.SparkEnv$.create(SparkEnv.scala:310) at > org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:186) at > org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:70) > > >
