NadeemMehraj opened a new issue #767: spark-submit: java.util.NoSuchElementException: spark.scheduler.mode URL: https://github.com/apache/incubator-hudi/issues/767 I am trying to setup Apache Hudi on an Ubuntu 16.04 server. I cloned the repo https://github.com/apache/incubator-hudi.git and then build it as `mvn clean install -DskipTests -DskipITs` The build completed successfully. I then proceeded with testing by following through https://hudi.incubator.apache.org/docker_demo.html When I try to execute `spark-submit --class com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class com.uber.hoodie.utilities.sources.JsonKafkaSource --source-ordering-field ts --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties --schemaprovider-class com.uber.hoodie.utilities.schema.FilebasedSchemaProvider` I am trying to setup Apache Hudi on an Ubuntu 16.04 server. I cloned the repo https://github.com/apache/incubator-hudi.git and then build it as mvn clean install -DskipTests -DskipITs The build completed successfully. I then proceeded with testing by following through https://hudi.incubator.apache.org/docker_demo.html When I try to execute `spark-submit --class com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class com.uber.hoodie.utilities.sources.JsonKafkaSource --source-ordering-field ts --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties --schemaprovider-class com.uber.hoodie.utilities.schema.FilebasedSchemaProvider` I get the output `... Exception in thread "main" java.util.NoSuchElementException: spark.scheduler.mode at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245) at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.SparkConf.get(SparkConf.scala:245) at com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:286) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ... ` I previously tried the version 0.4.7 and this issue did not arise in that version and I successfully executed the first 7 steps in hudi.apache.org/docker_demo.html. However, when I tried to execute the Step 8 (https://hudi.incubator.apache.org/docker_demo.html#step-8-schedule-and-run-compaction-for-merge-on-read-dataset) , trying to execute 'connect --path /user/hive/warehouse/stock_ticks_mor' resulted in error that there was no such command. So I went for the latest version.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
