Hey all, I'm trying to do some stuff with a YAML file in the Spark driver using SnakeYAML library in scala.
When I put the snakeyaml v1.14 jar on the SPARK_DIST_CLASSPATH and try to de-serialize some objects from YAML into classes in my app JAR on the driver (only the driver). I get the exception below. Yet when I dont have the snakeyaml jar on the SPARK_DIST_CLASSPATH but instead create a fat jar for my application (so it has the snakeyaml jar baked inside), then everything works fine. If I have both the jar on DIST_CLASSPATH & fat jarred (with sbt assembly) it still fails with the same exception. I'm guessing that SPARK_DIST_CLASSPATH has jars that end up in the 'primordial' class loader, because SnakeYAML is then live in this class loader, it cant find my application jar classes because they are loaded at a subsequent point/different class loader. What is the workaround for this? Thanks ~N Exception in thread "main" com.sai.cfg.InvalidConfigurationPropertyException at com.sai.cfg.ConfigurationParser$.getConfig(ConfigurationParser.scala:184) at com.sai.cfg.ConfigurationParser$.getGlobalConfig(ConfigurationParser.scala:171) at com.sai.cfg.ConfigurationParser.globalConfig$lzycompute(ConfigurationParser.scala:26) at com.sai.cfg.ConfigurationParser.globalConfig(ConfigurationParser.scala:26) at com.sai.cfg.ConfigurationParser.<init>(ConfigurationParser.scala:44) at com.sai.strategy.StrategyEngineMain$.delayedEndpoint$au$com$quantium$personalisation$strategy$StrategyEngineMain$1(StrategyEngineMain.scala:39) at com.sai.strategy.StrategyEngineMain$delayedInit$body.apply(StrategyEngineMain.scala:15) at scala.Function0$class.apply$mcV$sp(Function0.scala:40) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) at com.sai.strategy.StrategyEngineMain$.main(StrategyEngineMain.scala:15) at com.sai.strategy.StrategyEngineMain.main(StrategyEngineMain.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.yaml.snakeyaml.error.YAMLException: Class not found: com.sai.cfg.models.GlobalConfiguration at org.yaml.snakeyaml.constructor.Constructor.getClassForNode(Constructor.java:647) at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.getConstructor(Constructor.java:330) at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:340) at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:182) at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:141) at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:127) at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481) at org.yaml.snakeyaml.Yaml.load(Yaml.java:400) at com.sai.cfg.ConfigurationParser$.getConfig(ConfigurationParser.scala:181) ... 24 more