Sorry for the delay, you just did as I'm with the Azure CosmosDB (formerly DocumentDB) team. If you'd like to make it official, why not add an issue to the GitHub repo at https://github.com/Azure/azure-documentdb-spark/issues. HTH!
On Thu, May 11, 2017 at 9:08 PM ayan guha <guha.a...@gmail.com> wrote: > Works for me too....you are a life-saver :) > > But the question: should/how we report this to Azure team? > > On Fri, May 12, 2017 at 10:32 AM, Denny Lee <denny.g....@gmail.com> wrote: > >> I was able to repro your issue when I had downloaded the jars via blob >> but when I downloaded them as raw, I was able to get everything up and >> running. For example: >> >> wget https://github.com/Azure/azure-documentdb-spark/*blob* >> /master/releases/azure-documentdb-spark-0.0.3_2.0.2_2.11/azure-documentdb-1.10.0.jar >> wget https://github.com/Azure/azure-documentdb-spark/*blob* >> /master/releases/azure-documentdb-spark-0.0.3_2.0.2_2.11/azure-documentdb-spark-0.0.3-SNAPSHOT.jar >> spark-shell --master yarn --jars >> azure-documentdb-spark-0.0.3-SNAPSHOT.jar,azure-documentdb-1.10.0.jar >> >> resulted in the error: >> SPARK_MAJOR_VERSION is set to 2, using Spark2 >> Setting default log level to "WARN". >> To adjust logging level use sc.setLogLevel(newLevel). >> [init] error: error while loading <root>, Error accessing >> /home/sshuser/jars/test/azure-documentdb-spark-0.0.3-SNAPSHOT.jar >> >> Failed to initialize compiler: object java.lang.Object in compiler mirror >> not found. >> ** Note that as of 2.8 scala does not assume use of the java classpath. >> ** For the old behavior pass -usejavacp to scala, or if using a Settings >> ** object programmatically, settings.usejavacp.value = true. >> >> But when running: >> wget >> https://github.com/Azure/azure-documentdb-spark/raw/master/releases/azure-documentdb-spark-0.0.3_2.0.2_2.11/azure-documentdb-1.10.0.jar >> wget >> https://github.com/Azure/azure-documentdb-spark/raw/master/releases/azure-documentdb-spark-0.0.3_2.0.2_2.11/azure-documentdb-spark-0.0.3-SNAPSHOT.jar >> spark-shell --master yarn --jars >> azure-documentdb-spark-0.0.3-SNAPSHOT.jar,azure-documentdb-1.10.0.jar >> >> it was up and running: >> spark-shell --master yarn --jars >> azure-documentdb-spark-0.0.3-SNAPSHOT.jar,azure-documentdb-1.10.0.jar >> SPARK_MAJOR_VERSION is set to 2, using Spark2 >> Setting default log level to "WARN". >> To adjust logging level use sc.setLogLevel(newLevel). >> 17/05/11 22:54:06 WARN SparkContext: Use an existing SparkContext, some >> configuration may not take effect. >> Spark context Web UI available at http://10.0.0.22:4040 >> Spark context available as 'sc' (master = yarn, app id = >> application_1494248502247_0013). >> Spark session available as 'spark'. >> Welcome to >> ____ __ >> / __/__ ___ _____/ /__ >> _\ \/ _ \/ _ `/ __/ '_/ >> /___/ .__/\_,_/_/ /_/\_\ version 2.0.2.2.5.4.0-121 >> /_/ >> >> Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_121) >> Type in expressions to have them evaluated. >> Type :help for more information. >> >> scala> >> >> HTH! >> >> >> On Wed, May 10, 2017 at 11:49 PM ayan guha <guha.a...@gmail.com> wrote: >> >>> Hi >>> >>> Thanks for reply, but unfortunately did not work. I am getting same >>> error. >>> >>> sshuser@ed0-svochd:~/azure-spark-docdb-test$ spark-shell --jars >>> azure-documentdb-spark-0.0.3-SNAPSHOT.jar,azure-documentdb-1.10.0.jar >>> SPARK_MAJOR_VERSION is set to 2, using Spark2 >>> Setting default log level to "WARN". >>> To adjust logging level use sc.setLogLevel(newLevel). >>> [init] error: error while loading <root>, Error accessing >>> /home/sshuser/azure-spark-docdb-test/azure-documentdb-spark-0.0.3-SNAPSHOT.jar >>> >>> Failed to initialize compiler: object java.lang.Object in compiler >>> mirror not found. >>> ** Note that as of 2.8 scala does not assume use of the java classpath. >>> ** For the old behavior pass -usejavacp to scala, or if using a Settings >>> ** object programmatically, settings.usejavacp.value = true. >>> >>> Failed to initialize compiler: object java.lang.Object in compiler >>> mirror not found. >>> ** Note that as of 2.8 scala does not assume use of the java classpath. >>> ** For the old behavior pass -usejavacp to scala, or if using a Settings >>> ** object programmatically, settings.usejavacp.value = true. >>> Exception in thread "main" java.lang.NullPointerException >>> at >>> scala.reflect.internal.SymbolTable.exitingPhase(SymbolTable.scala:256) >>> at >>> scala.tools.nsc.interpreter.IMain$Request.x$20$lzycompute(IMain.scala:896) >>> at >>> scala.tools.nsc.interpreter.IMain$Request.x$20(IMain.scala:895) >>> at >>> scala.tools.nsc.interpreter.IMain$Request.headerPreamble$lzycompute(IMain.scala:895) >>> at >>> scala.tools.nsc.interpreter.IMain$Request.headerPreamble(IMain.scala:895) >>> at >>> scala.tools.nsc.interpreter.IMain$Request$Wrapper.preamble(IMain.scala:918) >>> at >>> scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$apply$23.apply(IMain.scala:1337) >>> at >>> scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$apply$23.apply(IMain.scala:1336) >>> at >>> scala.tools.nsc.util.package$.stringFromWriter(package.scala:64) >>> at >>> scala.tools.nsc.interpreter.IMain$CodeAssembler$class.apply(IMain.scala:1336) >>> at >>> scala.tools.nsc.interpreter.IMain$Request$Wrapper.apply(IMain.scala:908) >>> at >>> scala.tools.nsc.interpreter.IMain$Request.compile$lzycompute(IMain.scala:1002) >>> at >>> scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:997) >>> at scala.tools.nsc.interpreter.IMain.compile(IMain.scala:579) >>> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:567) >>> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) >>> at >>> scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807) >>> at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681) >>> at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37) >>> at >>> scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214) >>> at >>> org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37) >>> at >>> org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:94) >>> at >>> scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920) >>> at >>> scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) >>> at >>> scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) >>> at >>> scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97) >>> at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909) >>> at org.apache.spark.repl.Main$.doMain(Main.scala:68) >>> at org.apache.spark.repl.Main$.main(Main.scala:51) >>> at org.apache.spark.repl.Main.main(Main.scala) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at >>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) >>> at >>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) >>> at >>> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) >>> at >>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> sshuser@ed0-svochd:~/azure-spark-docdb-test$ >>> >>> >>> On Mon, May 8, 2017 at 11:50 PM, Denny Lee <denny.g....@gmail.com> >>> wrote: >>> >>>> This appears to be an issue with the Spark to DocumentDB connector, >>>> specifically version 0.0.1. Could you run the 0.0.3 version of the jar and >>>> see if you're still getting the same error? i.e. >>>> >>>> spark-shell --master yarn --jars >>>> azure-documentdb-spark-0.0.3-SNAPSHOT.jar,azure-documentdb-1.10.0.jar >>>> >>>> >>>> On Mon, May 8, 2017 at 5:01 AM ayan guha <guha.a...@gmail.com> wrote: >>>> >>>>> Hi >>>>> >>>>> I am facing an issue while trying to use azure-document-db connector >>>>> from Microsoft. Instructions/Github >>>>> <https://github.com/Azure/azure-documentdb-spark/wiki/Azure-DocumentDB-Spark-Connector-User-Guide> >>>>> . >>>>> >>>>> Error while trying to add jar in spark-shell: >>>>> >>>>> spark-shell --jars >>>>> azure-documentdb-spark-0.0.1.jar,azure-documentdb-1.9.6.jar >>>>> SPARK_MAJOR_VERSION is set to 2, using Spark2 >>>>> Setting default log level to "WARN". >>>>> To adjust logging level use sc.setLogLevel(newLevel). >>>>> [init] error: error while loading <root>, Error accessing >>>>> /home/sshuser/azure-spark-docdb-test/v1/azure-documentdb-spark-0.0.1.jar >>>>> >>>>> Failed to initialize compiler: object java.lang.Object in compiler >>>>> mirror not found. >>>>> ** Note that as of 2.8 scala does not assume use of the java classpath. >>>>> ** For the old behavior pass -usejavacp to scala, or if using a >>>>> Settings >>>>> ** object programmatically, settings.usejavacp.value = true. >>>>> >>>>> Failed to initialize compiler: object java.lang.Object in compiler >>>>> mirror not found. >>>>> ** Note that as of 2.8 scala does not assume use of the java classpath. >>>>> ** For the old behavior pass -usejavacp to scala, or if using a >>>>> Settings >>>>> ** object programmatically, settings.usejavacp.value = true. >>>>> Exception in thread "main" java.lang.NullPointerException >>>>> at >>>>> scala.reflect.internal.SymbolTable.exitingPhase(SymbolTable.scala:256) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$Request.x$20$lzycompute(IMain.scala:896) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$Request.x$20(IMain.scala:895) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$Request.headerPreamble$lzycompute(IMain.scala:895) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$Request.headerPreamble(IMain.scala:895) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$Request$Wrapper.preamble(IMain.scala:918) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$apply$23.apply(IMain.scala:1337) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$apply$23.apply(IMain.scala:1336) >>>>> at >>>>> scala.tools.nsc.util.package$.stringFromWriter(package.scala:64) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$CodeAssembler$class.apply(IMain.scala:1336) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$Request$Wrapper.apply(IMain.scala:908) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$Request.compile$lzycompute(IMain.scala:1002) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:997) >>>>> at scala.tools.nsc.interpreter.IMain.compile(IMain.scala:579) >>>>> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:567) >>>>> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) >>>>> at >>>>> scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807) >>>>> at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681) >>>>> at >>>>> scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395) >>>>> at >>>>> org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38) >>>>> at >>>>> org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37) >>>>> at >>>>> org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37) >>>>> at >>>>> scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214) >>>>> at >>>>> org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37) >>>>> at >>>>> org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:94) >>>>> at >>>>> scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920) >>>>> at >>>>> scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) >>>>> at >>>>> scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) >>>>> at >>>>> scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97) >>>>> at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909) >>>>> at org.apache.spark.repl.Main$.doMain(Main.scala:68) >>>>> at org.apache.spark.repl.Main$.main(Main.scala:51) >>>>> at org.apache.spark.repl.Main.main(Main.scala) >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> at >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>> at >>>>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) >>>>> at >>>>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) >>>>> at >>>>> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) >>>>> at >>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) >>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>>>> sshuser@ed0-svochd:~/azure-spark-docdb-test/v1$ >>>>> >>>>> I think I am missing some basic configuration here or there is >>>>> classpath related issue. Can anyone help? >>>>> >>>>> Additional info: >>>>> Environment: HDInsight 3.5, based on HDP 2.5 >>>>> >>>>> sshuser@ed0-svochd:~/azure-spark-docdb-test/v1$ echo $JAVA_HOME >>>>> /usr/lib/jvm/java-8-openjdk-amd64 >>>>> >>>>> sshuser@ed0-svochd:~/azure-spark-docdb-test/v1$ echo $SPARK_HOME >>>>> /usr/hdp/current/spark2-client >>>>> >>>>> sshuser@ed0-svochd:~/azure-spark-docdb-test/v1$ java -version >>>>> openjdk version "1.8.0_121" >>>>> OpenJDK Runtime Environment (build >>>>> 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) >>>>> OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode) >>>>> >>>>> sshuser@ed0-svochd:~/azure-spark-docdb-test/v1$ uname -a >>>>> Linux ed0-svochd 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 >>>>> UTC 2017 x86_64 x86_64 x86_64 GNU/Linux >>>>> sshuser@ed0-svochd:~/azure-spark-docdb-test/v1$ >>>>> >>>>> -- >>>>> Best Regards, >>>>> Ayan Guha >>>>> >>>> >>> >>> >>> -- >>> Best Regards, >>> Ayan Guha >>> >> > > > -- > Best Regards, > Ayan Guha >
