Right, we should remove option 1 from doc, as it is not standard way to adding additional jar for spark.
Created https://jira.apache.org/jira/browse/ZEPPELIN-4130 Paul Brenner <pbren...@placeiq.com> 于2019年4月23日周二 上午8:55写道： > Amazing. Using spark.jars.packages instead of the dependencies section of > the interpreter completely fixes it. Thanks for the help! > > I noticed that the method I was originally using is listed as option #1 in > the docs (here > <http://zeppelin.apache.org/docs/0.8.0/interpreter/spark.html#dependency-management>). > Does it make sense to include a note that options 2 and 3 are the > “standard” way and may succeed when method #1 fails? Or is that going into > unnecessary detail for the docs or not even true in most cases? > > > > PAUL BRENNER > > Sr. Data Scientist > pbren...@placeiq.com | (217) 390-3033 | www.placeiq.com > twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq > <https://www.linkedin.com/company/placeiq/> > [image: PlaceIQ is named an Oracle Data Cloud Premier Data Provider] > <https://www.placeiq.com/2019/04/placeiq-named-a-premier-data-provider-and-the-sole-location-data-company-in-official-launch-of-oracle-data-cloud-program/> > [image: PlaceIQ: Oracle Data Cloud Premier Data Provider] > <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00> > On Apr 21, 2019, 10:28 AM -0400, Jeff Zhang <zjf...@gmail.com>, wrote: > > Hi Paul, > > I could not reproduce this issue in branch-0.8 (0.8.2) even using your > configuration( spark 2.4, yarn-client, add dependency via zeppelin > interpreter dependencies interface) > > One thing you can try is to add dependency via setting spark.jars.packages > which is the standard way to set additional jars/packages. > > > > > Paul Brenner <pbren...@placeiq.com> 于2019年4月21日周日 下午9:23写道： > >> >> 1. I used Spark 2.4.0 >> 2. I used yarn-client mode >> 3. Dependency is added via zeppelin’s interpreter dependencies >> interface. >> >> Thanks for looking into this! We still don’t have a working fix in place >> so are held back from migrating to the wonderful and glorious features of >> 0.82. >> >> >> >> PAUL BRENNER >> >> Sr. Data Scientist >> pbren...@placeiq.com | (217) 390-3033 | www.placeiq.com >> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq >> <https://www.linkedin.com/company/placeiq/> >> [image: PlaceIQ is named an Oracle Data Cloud Premier Data Provider] >> <https://www.placeiq.com/2019/04/placeiq-named-a-premier-data-provider-and-the-sole-location-data-company-in-official-launch-of-oracle-data-cloud-program/> >> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider] >> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00> >> On Apr 21, 2019, 1:44 AM -0400, Jeff Zhang <zjf...@gmail.com>, wrote: >> >> Hi Paul, >> >> Could you let me know the following questions ? >> >> 1. Which spark version do you use ? >> 2. Which mode do you use ? local / yarn /standalone >> 3. How did you add dependency ? via spark.jars.package or via zeppelin's >> interpreter dependencies ? >> >> Paul Brenner <pbren...@placeiq.com> 于2019年4月20日周六 下午8:00写道： >> >>> Yes. >>> >>> Add com.databricks:spark-avro_2.11:3.2.0 as a dependency to a spark >>> interpreter then try using that interpreter to run any code in a notebook >>> to reproduce. >>> >>> For our internal library that was causing it I went through and tried >>> excluding everything that that library imported. It turns out it was >>> importing org.scala-lang:scala-library. Excluding that may have resolved >>> the issue for our internal package. >>> >>> >>> >>> PAUL BRENNER >>> >>> Sr. Data Scientist >>> pbren...@placeiq.com | (217) 390-3033 | www.placeiq.com >>> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq >>> <https://www.linkedin.com/company/placeiq/> >>> [image: PlaceIQ is named an Oracle Data Cloud Premier Data Provider] >>> <https://www.placeiq.com/2019/04/placeiq-named-a-premier-data-provider-and-the-sole-location-data-company-in-official-launch-of-oracle-data-cloud-program/> >>> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider] >>> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00> >>> On Apr 20, 2019, 3:30 AM -0400, Jeff Zhang <zjf...@gmail.com>, wrote: >>> >>> Hi Paul, >>> >>> Could you describe how to reproduce it ? >>> >>> Paul Brenner <pbren...@placeiq.com> 于2019年4月20日周六 上午12:30写道： >>> >>>> I’m trying to move my company up from zeppelin 0.8 to 0.81 or 0.82. >>>> However, we find that when including certain dependencies on the newer >>>> versions of zeppelin spark interpreters won’t start and instead throw the >>>> error I’ll paste below. This is if we have zeppelin.spark.useNew set >>>> to true. Setting it to false just gives a null pointer exception instead. >>>> An example of a dependency that causes this is >>>> com.databricks:spark-avro_2.11:3.2.0, >>>> but there are also some internal packages we use that we haven’t been able >>>> to find anything that should be causing this conflict. >>>> >>>> I found someone with a similar problem on ZEPPELIN-4074 >>>> <https://jira.apache.org/jira/browse/ZEPPELIN-4074> but no one seems >>>> to have any idea what to do. Any ideas? It sounds like isJavaAtLeast got >>>> fixed to work with JDK 9 in scala 12 (see here >>>> <https://github.com/scala/scala/pull/5276>), but who invited JDK 9 to >>>> this party? I certainly didn’t. >>>> >>>> Zeppelin 0.8 is really struggling for us so we are desperate to upgrade >>>> but this is holding us back, would really appreciate any help at all! >>>> >>>> java.lang.NumberFormatException: Not a version: 9 at >>>> scala.util.PropertiesTrait$class.parts$1(Properties.scala:184) at >>>> scala.util.PropertiesTrait$class.isJavaAtLeast(Properties.scala:187) at >>>> scala.util.Properties$.isJavaAtLeast(Properties.scala:17) at >>>> scala.tools.util.PathResolverBase$Calculated$.javaBootClasspath(PathResolver.scala:276) >>>> at >>>> scala.tools.util.PathResolverBase$Calculated$.basis(PathResolver.scala:283) >>>> at >>>> scala.tools.util.PathResolverBase$Calculated$.containers$lzycompute(PathResolver.scala:293) >>>> at >>>> scala.tools.util.PathResolverBase$Calculated$.containers(PathResolver.scala:293) >>>> at scala.tools.util.PathResolverBase.containers(PathResolver.scala:309) at >>>> scala.tools.util.PathResolver.computeResult(PathResolver.scala:341) at >>>> scala.tools.util.PathResolver.computeResult(PathResolver.scala:332) at >>>> scala.tools.util.PathResolverBase.result(PathResolver.scala:314) at >>>> scala.tools.nsc.backend.JavaPlatform$class.classPath(JavaPlatform.scala:28) >>>> at scala.tools.nsc.Global$GlobalPlatform.classPath(Global.scala:115) at >>>> scala.tools.nsc.Global.scala$tools$nsc$Global$$recursiveClassPath(Global.scala:131) >>>> at scala.tools.nsc.Global$GlobalMirror.rootLoader(Global.scala:64) at >>>> scala.reflect.internal.Mirrors$Roots$RootClass.<init>(Mirrors.scala:307) at >>>> scala.reflect.internal.Mirrors$Roots.RootClass$lzycompute(Mirrors.scala:321) >>>> at scala.reflect.internal.Mirrors$Roots.RootClass(Mirrors.scala:321) at >>>> scala.reflect.internal.Mirrors$Roots$EmptyPackageClass.<init>(Mirrors.scala:330) >>>> at >>>> scala.reflect.internal.Mirrors$Roots.EmptyPackageClass$lzycompute(Mirrors.scala:336) >>>> at >>>> scala.reflect.internal.Mirrors$Roots.EmptyPackageClass(Mirrors.scala:336) >>>> at >>>> scala.reflect.internal.Mirrors$Roots.EmptyPackageClass(Mirrors.scala:276) >>>> at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:250) at >>>> scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:73) at >>>> scala.tools.nsc.Global.rootMirror(Global.scala:71) at >>>> scala.tools.nsc.Global.rootMirror(Global.scala:39) at >>>> scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:257) >>>> at >>>> scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:257) >>>> at >>>> scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1390) >>>> at scala.tools.nsc.Global$Run.<init>(Global.scala:1242) at >>>> scala.tools.nsc.interpreter.IMain.scala$tools$nsc$interpreter$IMain$$_initialize(IMain.scala:139) >>>> at scala.tools.nsc.interpreter.IMain.initializeSynchronous(IMain.scala:161) >>>> at >>>> org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:85) >>>> at >>>> org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102) >>>> at >>>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62) >>>> at >>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) >>>> at >>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616) >>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at >>>> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140) >>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>> at java.lang.Thread.run(Thread.java:748) >>>> >>>> >>>> >>>> PAUL BRENNER >>>> >>>> Sr. Data Scientist >>>> pbren...@placeiq.com | (217) 390-3033 | www.placeiq.com >>>> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq >>>> <https://www.linkedin.com/company/placeiq/> >>>> [image: PlaceIQ is named an Oracle Data Cloud Premier Data Provider] >>>> <https://www.placeiq.com/2019/04/placeiq-named-a-premier-data-provider-and-the-sole-location-data-company-in-official-launch-of-oracle-data-cloud-program/> >>>> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider] >>>> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00> >>>> >>> >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >>> >>> >> >> -- >> Best Regards >> >> Jeff Zhang >> >> > > -- > Best Regards > > Jeff Zhang > > -- Best Regards Jeff Zhang
